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• ' FOREWORD 

This report prepared in partial fulfillment of the requirement of 
Technical Development Plan (TDP) P43-03 (POIA),. considers decision making 
at the top management levels of the Naval Education' and Training Command. 
Traditionally top management decision making relies heavily on informal 
techniques — intuitive judgments by experienced managers, traditions and 
rules of thumb" peculiar to each organization, or'^'nformal consensus 
among a group of managers. Currently, however, the training command must 
do a job which is unprecedented in its complexity and must do it despite 
declining resources. These considerations indicate the desirability of 
increased effectiveness in'decision making. This report surveys the 
potential role of and desirability of more formal approaches to some top 
management decision problems. 

Appreciation is expressed to the staffs of the organizations studied 
for their cooperation. Special thanks are due to Mr. E. Riley (Chief of 
Naval Education and Training Staff) and Mr. I. McNaney of the Training 
Analysis and Evaluation Group who coordinated the field research effort. 
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section i 
"■ \ntroduction 
a. background . " 

In 1966 the Chief of Naval Operations promulgated the Advanced Devel- 
opment Objective (ADO) 43-03X', "Education and Training Development." This 
document recognized both the importance of the education and training 
methods and management techniques in making training more effective. 
Subsequently, Technical Development Plan (TDP) P43r03X was prepared in 
response to the ADO. Part OlA of the TDP, "Design of Training Systems," 
addressed the management of the training effort, with the goal of providing 
training managers at all levels witn an effective decision making capabil- 
ity. Implementation of this subproject is the responsibility of the Train- 
ing Analysis and Evaluation Group (TAEG), Orlando, Florida. 

The TAEG has contracted with the Federal Systems Division of Interna- 
tional Business Machines (IBM) to conduct a study in support of this work. 
IBM has, among other things, conducted a broad survey of management func- 
tions within ' the Naval Education and Training System, and proposed a number, 
of areas in which computerized decision aids would be useful to training 
managers. Initially, three computer models were designed and developed,, all 
of which address problems of managing student flows, course and resource 
planning, and system requirements. A fuller discussion of this background 
can be found in a series of reports prepared by TAEG, particularly TAEG 
Reports No. 11-1 (Lindahl , et al., 1973), 11-2 (Lindahl ana Gardner, 1974), 
12-1 (Volumes I and II), and 12-2 (Volumes I, II and III). 

B. PURPOSE 

The purposes of this report are: 

(1) to provide training managers basic but pertinent reference 
material on the subject of decision analysis, (2) to stimulate interest 
in decision analysis for the purpose of adopting the concepts in problem 
solving, and (3) to present a consultant's observation and- recommendation 
for improving decision making within the Naval Education and Training 
Coimand (NAVEDTRACOM) . 

C. REPORT ORGANIZATION . - 

In section II the process of decision making in large organizations 
is discussed. Included are some notes on the decision making mechanisms 
observed in large organizations, a taxonomy of decision making, an intro- 
duction to the field' of decision theory, and a summary of the use of 
decision theory in organizations. In section III the background and 
responsibilities of the Chief of Naval Education and Training (CNET) 
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organization eire presented preparatory to a discussion of the decision 
making process observed in CNET, Chief of Naval Education and Trainina Support 
(CNETS), and Chief of Naval Technical Training (CNTECHTRA). Finally/ 
section IV consists of a sunmary and recommendations for improvement in 
the NAVEDTRACOM decision making process. 
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SECTION II 

DECISION MAKING IN LARGE ORGANIZATIONS 

* "One of the purposes of this •report is to suggest improvements to the 
decision'making process within the NAVEDTRACOM. It is worth ^.oting that 
formal techniques, whether they go under the guise of computerized manage- 
ment information systems, operations research, decision theory, etc., have 
a disappointing track record in general management, although there have 
been some outstanding successes. Part of the reason for this may lie i-n 
the way decisions tend to get made in organizations, whether the methods 
used be formal or informal. 

A. ORGANIZATIONAL GOAL SETTING AND DECISION MAKING .. • 

In traditional economic theory, busin^ess firms aqt in such a wety as 
to maximize profits, given th,at there are Sfehavioral c^oj^straihts imposed 
by the competition, the marketplace, government, and society. The picture,^ 
of decision making in this theory i.s 'nat of managers at]d ofgairrz^-Qnal 
forces together working to pull together information., assess uncertainty, 
and make rational choices to maximize profit under uncertainty. One would 
predict, under these assumptions, a number of behavioral patterns which 
do not seem- to exist in real organizations; further, the traditional theory 
does not provide for a good many important behavioral patterns observable in 
actual firms. Traditional theory doesn't allow^A for example, for complex 
organizations, problems of control, standard prafctices and operating pro- 
cedures, budgets, ^poti tics, ^ entrenched "bosses'' or aslDiring ''middle 
managers."' In short, the traditional theory of the firm does not address 
the mechanism by which managerial decisions are made. It is precisely 
this mechanism, however, which is the focus of this research. Thus, it is 
vital to have a good understanding of organizational decision making. This 
would not only give^a basis for analyzing NAVEDTRACOM, bBt- also provide a 
realistic background for evaluating suggested improvements. A standard 
reference in the* descriptive theory of organizational decision making is 
Cyert and March (1963). * This work has "an explicit emphasis on the actual 
process of organizational decision making as its basic, research commitment." 
The remainder of this section draws heavily upon the framework and ob5,erva- 
tions made in this book. Although the Cyert and March research was done 
specifically on large, multiproduct, profit-making firms in oligopolistic ^ 
markets, the major observations on organizational decision mechanisms seem 
to be valid, for any large organization, including public sector not-for- 
profit organizations, 

ORGANIZATIONAL GOALS 

Any organization is a coalition of individuals, each with his own set 
of goals. -These individuals are, in turn, organized into groups, the groups 
into larjger groups, and so on, with each level adding a new set of goals. 
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Hew then d&es one manage to set goals for the enti.re organi za-^ion withaut 
causing fatal conflict? Jhere are two ^inswers to this question: first\ 
the goals themselves are structured in such. a way as to avoid conflict; 
and second, the goal setting mechanism -allows for -conflict resolution. 

Goals are rarely structured in precise, unambiguous terms. On the 
contrary, st.udies of organizational objectives suggest that agreement on 
.objectives usually exists only when objectives" are highly ambiguous, and" 
that behind this agreement lies considerable uncertainty and disagreement 
on subgoals. To a large extent critical' areas of conflict are never . 
brought into the' open, and agreeing, in essence, to disagree is usually a' 
semipermanent way of life. Goals which are set usually take the form of 
aspiration levels, which are subject, to, change over time, rather than 
imperatives to "maximize" or "minimize.", Thrs tends to make confli-ct 
resolution less urgent, as organizational units ara. judged by the degree 
to which they attain the aspiration levels, which are largely set- accord- 
ing-to past experience, rather than by what is theoretically possible.., 

Th'e goaVsetting mechanism is also an important stabilizing influence 
Organizational goals are riot, as a rule, set through a process of analytic 
determination of optimal procedures, but through a bargaining process. 
The parties to the goal-setting reach agreements through sharing rewards, 
.not only such monetary rewards as budgets, salaries, and contracts, but 
nonmonetary rewards, as well: promotions, "citations, policies, personal 
treatments, private commitments, etc. This pracess allows parties to 
settle conflicts through relatively non-disruptive mechanisms. The ob- 
jectives resulting from this process have a number of important attributes 
(Cyert and March, 1963, p. 32): 

... 1. They are imperfectly rationalized. The exact forms of the agree 
ments will depend at leasfas much on the bargaining skills of the leaders 
the history of the bargaining session, and the particular resource scar- 
• cities faced a^ on the merits of the issuer themselves-. 

2. Some objectives are stated in the forni of aspiration-level com- 
mitments, such as "We must allocate ten percent of our total budget 'to 
research." The determination of the "ten. percent" undoubtedly has little 
to_ do with co$t/benefit analyses of the particular research projects avaii 
able, but a good deal "to do wit>^ the prestige, power, and persuasiveness 
of the research director. 

3. Some Objectives are stated in- a nonoperational form'^ Such ob- 
jectives have the advantage of being consistent with virtually any set of 
operational objectives. For' example, nonoperational goals are evidenced 
in such political rhetoric as "We must 'crack down.' on crimej while simul- 
taneously protecting citizens rights and eliminating police brutality." 
This can be cited in favor of such diverse operational goals as increasing 
the use of wiretapping and eliminating the use of wiretapping. • 
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Once organizational objectives have been esta:blished, a number of 
stabilizing mechanisms tend to make thenfi permanent. Budgets are based 
partly on past budgets. Allocations of functions and other ' precedents 
are remembered and followed, sometimes taking the form of "rules of thumb" 
jr "the usual way we do things," and occasionally becoming part of the 
standard operating procedure. Thus accidents of organizational geneology- 
tend to become perpetuated, to be abandoned only under pressure. Frrther, 
organizations tend to build up a certain amount of "fa t"^; (termed "organiza- 
tionaV sWck" by Cyert and March, 1963). In case of adversity, this prx)- 
vides a cushion so that business as usual can continue, perhaps indefini- 
tely or at least until a. new bargaining cycle can be completed. 

It is worth contrasting the preceding description of organizational 
goal ^setting with an objective, or "rational" procedure, v/hich would re- 
quire the fT'nn to predict, the environment, survey all possible actions, 
then pick a set of goals which would result in the best actions. It is 
not at all clear that the objective procedure, even if it, were possible 
to implement it, woiJld be the better way to choose goals. Clearly the 
actual process contains valuable mechanisms fov keeping the organization 
'stable and viable, and substituting the "rational" procedure could easily 
result in the surfacing of chaotic conflict. 

ORGANIZATIONAL EXPECTATIONS . . 

Expectations are seen as the result of- drawing inferences from avail- 
able information. Thus, while goal setting might be, seen as both a way 
of stating what considerations are important , in an organization and of 
setting up evaluation standards expectations might be thought of as pro- 
jections of whdt will happen. ^One can study expectations, then., both from 
the standpoint of how inferences are drawn and of how information is made 
available in the organization. 

• ^On the subject of inference drawing, two general observations can be.'^ 
made. FirstV expectations are biased, both consciously and unconsciously, 
by hopes, wishes ,^ goals , and internal bargaining. The manipulation is 
usually subtle,' but is occasionally overt. A classic case of the latter 
occurred in a major Naval weapons system acquisition procurement project, 
which suffered la budget cut severely limiting the number of units to be 
purchased" for the fleet. The cut was justified by a study showing that the 
new lower number of weapons (less than half thenumber originally planned) 
was adequate to meet the needs o*f the fleet; however the study was in 
response to, rather than in anticipation of, the budget cut. Second, it 
app'^ars that the .computational power and precision available i'n organiza- 
tions are limited^ One observes that only on a few of the potentially 
relevant variables are data gathered and projections made*. Also, the pro- 
jections that are made tend to be very simplistic, requiring .a minimum of 
calculation. There is, of course, nothing wrong with simplicity, but by 
and large the capacity for more sophisticated. methods seems to be lacking, 
even when such methods may be justified. 
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Information availability within an organization is strongly affected 
by the nature of the data gathering system. Information becomes available 
in a' fragmented, sporadic fashion at different organizational and geo- 
graphic locations. Communication of the information is subject to s,evere 
bias, delay, and filtering effects; internal communication is, in fact, 
a significant competitive weapon within the organization- Thus the com- 
munication system introduces significant distortion into the system; over 
the long run, however, systematic bias seems to be at least partially de- 
tected and accounted for. 

Most significant decisions require information which is not readily 
available, thus implying the need for a search procedure. In rational 
economic theory, a firm would have a portfolio of potential investments, 
against which new proposals -would be continually evaluated. Information 
search should, in theory^ be treated^^just like any other proposal, as a 
potential investment of resources which is expected to yield a benefit. 
An information search project, if accepted, would be analogous to a 
prospecting expedition; the world is searched in a systematic way with 
particular data needs in mind, and any nuggets of information found are 
brought back and assayed. In reality, this ideal picture is inaccurate 
on a number of counts. First, project evaluation and information search 
do not occur continually, but only as a result of fairly obvious problems. 
Organizations do not plan nearly 'as much as they fight fires, reacting to • 
current crises. Second, search activity is not itself treated as an 
investment. Rather, there are various levels of search activity that are 
called into play, so that for a given situation there is a standard search 
procedure. Further, the criterion of search activity is feasibility rather 
than opi:imality. As soon as something is found that-seems to (more or less) 
solve the problem, the search stops; the only questions asked are "Is it 
feasible?" and ''Is it better than what we have now?," rather than "Is this 
the best possible way of handling the problem?" Finally, the search pro- 
cedure is not nearly so. much a prospecting expedition as a "mating dance." 
The direction of the search is largely determined by the conspicuousness 
of the alternatives, and as various people, both inside and outside the 
organization, have their ow.n interests tied up in the decision they natu- - 
rally try to make their preferred alternatives the most conspicuous ones. 
Thus, the organization is not only in search of information, but interest 
groups are trying to make at least parts of ,^the -information known to the 
organization. , . v 



ORGANIZATIONAL CHOICE 




There are. three basic principles in decision making: (1) avoid con- 
.slderation of uncertainty, (2) maintain organizational rules and pt?ecedents, 
and (3) keep decision rules simple. • * 

^ ; It seems paradoxical to speak of firms avoiding the consideration of 
uncertainty. After all, the world is uncertain. There ai'e procedures. 
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however, which minimize the need to predict uncertain events. First, firms 
do very little meaningful long range planning ^ fhoving instead from one 
crisis to the next. Second, firms rely heavily upon standard rules for 
doing things, whether these be traditional methods, general industry prac- 
tice, or standard operating procedures. These not only influence (and in' 
many cases dictate) the decisions which are made, but provide stability and 
predictability to the orga nization. Thus, when Department A is working on 
a problem, it is already known wlTaT^tlT5^HsiJ0°i^es^ ef-Bepa^^man±s^B and C 
to their parts of the same problem are going to be. In addition, planning 
records mad,e within the organization act to fix commitments and expectations. 
"Plans, like other standard operating procedures, reduce a complex world to 
a somewhat simpler one. Within rather large limits, the organization sub- 
stitutes the plan for the world--partly by making the world conform to the 
plan, partly b^' pretending it does" (Cyert and March, 1963, p. 111). When , 
possible, organizations arrange a negotiated env'ronmerit. 

The stahiifi-zing influence of .standard procedures would largely be lost 
if the procedures were to change frequently. The procedures build up around 
themselves a myriad of precedents, understandings, and unspoken connotations, 
thus becoming entrenched. When procedures are changed there is always a 
period of uncertainty and unsettlement until things "get worked out" again. 
The second principle of decision making, then, is to maintain the organiza- 
tional procedures and precedents. 

. The third and final principle is to keep decision rules simple. Gen- 
erally one searches for feasible alternatives (rather than optimal ones) 
and implements the first one encountered. Thus the search procedures 
strongly affect the decision making process. There are, of course, prob- 
lems which come up which are not adequately covered by -standard procedures. 
Rather than elaborate procedures to cover a wider variety of problems, 
organizations opt to keep the rule simple and rely on individual judgment 
to provide flexibility. 

SUMMARY 

It is important to recognize the potential of analytic techniques. 
First, on problems that are reasonably complex, particularly when uncer- 
tainty is involved, formal techniques can, rf used properly, nearly always 
result in better decisions. Second, effective use of formal techniques does 
not require replacement of the entire corporate decision making apparatus. 
An individual manager can make effective use of such techniques on problems 
falling within his own area, or in broader problems, to yield results for 
his own evaluation of alternatives, or as ammunibion in corporate give and 
take. The techniques themselves are useful for broad or narrow problems, 
or for top or lower level decisions; however, due to the fact that at some 
•levels, for some problems , explicit rationality would be a positive 
■hinderance rather than a help, one should pick, the problems to be analyzed 
rather carefully. . • • ' 
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•B, DECISION MAKING - A TAXONOMY 

There are any number of ways of classifying deci sions--by subject area 
(inventory decisions, personoel decisions, ,etcj, by managerial level (top 
level decisions, middle level decisions, lower level decisions), by im- 
portance (critical, major, minor, et^f.), and so on. A taxonomy should, 
though, be an aid to good decision making, not simply an arbitrary classi- 
-4A-^ i on~sd^me^^— 14©^-^^^-^ _a-^X0nomy -no-fe-Gn4y — - 

to attach a classification label to^a given decision problem, but to find 
an approach useful in solving his problem. Thus, the starting point in- 
constructing a decision taxonomy is to consider- the decision making process 
itself. 

THE DECISION MAKING PROCESS ' 

Decision making involves foiir steps: forecasting or projecting, 
identifying alternatives, determining measures of effectiveness of possible 
outcomes, then making the choice. The first step, forecasting or projecting 
the needs of the organization, the outside environment, future constraints 
upon the organization, and so forth, defines the context against which any 
decision must be evaluated. In some decisions this step is critical; deter- 
mination of capacity needs in training facilities depends on forecasts of 
training volume and methodology, for example. There are other cases where 
this step is less important, as in choosing among methods to present a 
standard block of training. In forecasting, it seems useful to distinguish 
two extreme situations. The first extreme, called in this paper a "well 
defined" situation, is one where the objectives, the constraints, the 
structure, and the. ralationships among' variables in the problem are 
relatively well understood. A good example of this is an inventory policy 
decision. Demand is well understood and, though uncertain, amenable to 
analysis. Costs^of carrying a given inventory level can be defined and 
calculated. Costs of a stackout, though harder to calculate, can still be 
understood and approximated. Relationships among supply, demand, stock 
level, lead times, etc., can be easily specified. Contrast this with the 
other extreme, an "ill defined" situation, such as a decision problem like 
"what should CNET's policy toward enlisted personnel training be over the 
next five years?" Here a good deal of effort must be expended simply in 
defining the critical questions, the alternatives, and relevant considera- 
tions before the problem can be meaningfully discussed. Initially, at 
least, the ill defined problem would seem to call for a different type of 
approach than the well defined problem. 

. Another characteristic of decision problems which can be observed 
during the forecasting phase is the importance of uncertainty. Occasionally 
the uncertainty in a problem is relatively minor, so one can act as if all 
relevant factors were known. Many resource allocation problems and sched- 
uling problems, for example, are of this type. Most real problems, 
however, involve uncertainty to a major degree, and decisions made under 
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the assumption of certainty may be grossly misleading. Thus, one must 
adopt different techniques for dealing with these two types of problems. 

The second step in decision making involves specifying alternatives. 
In well defined problems these are usually readily apparent; in the inven- 
tory policy decision, alternatives are defined by all possible stock 
levels and reordering polic ies, and t he choice of the inventory, control 
■^mechOTi-sm tt^^ iTi "ITl^eti ned problems a major effort may be needed 
to define a set of reasonable alternatives,' If uncertainty is a considera- 
tion, then it may be necessary to specify contingent, as well as immediate 
alternatives. Thus, techniques for dealing with decision problems under 
uncertainty must include methods of identifying and describing contingent 
decision structures. 

In the third step in the decision process, measures of effectiveness 
are specified. This involves considering the job to be done and identify- 
ing considerations relevant in evaluati.ng alternatives. Sometimes a single 
measure can be identified as an overriding consideration, but more often 
one must deal with multiple criteria, some of which may not be measurable. 
Suppose, for example, that the job is to train pilots. Some relevant con- 
siderations are the length and cost of training, final pilot proficiency, 
capacity of the training pipeline, and a number of others. Only some of 
these considerations are directly measurable, so it is necessary to specify 
ways of estimating nonmeasurable outcomes. An approximate measure of 
pilot proficiency, for example, can be constructed by use of r.ubjective 
evaluations, such as instructor comments, and results of quan:i tative 
tests^ such as proficiency exams. Finally, these various measures are 
combined into a single measure of effectiveness. As a rule this last step 
is not performed explicitly; it is unusual to find a manager who has speci- 
fied in any coherent manner the kinds of trade offs hq is willing to 
make among effectiveness ^measures. Usually decisions are made on a case- 
by-cas.e basis by consensus or "common sense"; these intuitive decisions do^ 
however, define implicit trade off structures. 

V ^ The final step in decision making is to select the best alternative 
amon^' those specified. A variety of techniques, ranging from snap judg- 
ments to sophisticated and expensive computel^ized models are available 
to aid in this step, A major purpose of this taxonomy is to wed the 
decision problem with the appropriate technique. 

THE TAXONOMY . 

The taxonomy developed in this research is outlined in figure 1. 
Branches on the breakdown tree in the figure are numbered, so that the 
discussion can be keyed to the appropriate part of the breakdown struc- 
ture. A breakdown, between problems not requiring tfiought (branch [1]) 
and those requiring thought (branch [2]), is introduced at the top of the 
breakdown structure. This is introduced, somewhat facetiously, as a 
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reminder! that, many problems are too trivial, too obvious, or too'* con- 
strained! to justify detailed analysis. It is important to rememjber that 
the scop^ and expense of the analysis and the magnitude of the original 
problem must be kept in proportion. Most- of the solution methods dis- 
cussed below can be applied in either relatively simple or relatively ' 
complex manners, depending on the demands of the problem. 

WELL DEFINED, CERTAIN , .UNIDIMENSIONAL PROBLEMS (BRANCHES [3] and [4]). In 
discussing the taxonomy, consider first a problem which (1) requires 
thought, (2) is well /defined, (3) can be treated as. certain, and (4) has 
a single measure of Effectiveness. Most real problems aren't this uncom- - 
nlicated, but this simplest case is important as an approximation for 
many real problems and as "an introduction to the more complex .situations. 
Nearly all solution techniques for more complex problems consist of ways 
to reduce them to this simplest case, then using the techniques for this 
case to solve the problem. An enormous variety of techniqi^fes suited for . 
such problems is known; here they are classified broadly as simulation 
techniques (branch [3]) and optimization techniques (branch [4]); Some 
of these solution techniques require computers for useful implementation. 

Simulation techniques are projection techniques. Given a particular 
decision, the. siniiulation method predicts the outcome. In order to. deter- 
mine the best decision, one must redo the simulation many times, trying all 
possible decisions in order to find the best one. Informal, "seat-of-the- 
pants" '[judgments are classified as simulation techniques in this report, as 
judgments consist of projecting the consequences of an action,' then choosing 
among the actions. Simulation techniques can be extremely straightforward. 
Every manager who has projected a cash flow, forecasted expenses, or set up a 
budget hasy; in effect, performed a simulation. The idea in Simulation is to 
formulate set of rules which govern the behavior of a system, then apply 
those rilles to see how the system^ acts. The more complex the simulation be- 
comes, the more expensive, time consuming, and error prone it becomes, but th 
more potential usefulness it has. There have been many cases where comple)^; 
simulat^ions yield insights impossible to obtain in any other way. 

/ ■ . ■ . ' ••" 

(Optimization techniques differ from simulation techniques in that 
optimization techniques are designed to not only predict outputs, but to 
determine automatically the best possible decision. The price one pays for 
this additional feature is. usually a good deal of additional complexity. 
There are a number of "standard" types of problems (such as linear pro- 
gramming models) which can be relatively easily solved. If a particular 
problem fits one of these types, then optimizing may be straightforward, 
although perhaps expensive. 

A complete discussion of optimization is well beyond the scope of this 
report. An introduction to this field can be found in Wagner (1969). A 
good, elementary, managerial-oriented discussion of optimization and simu- 
J^-ti-on--y^j±_ej^mplj^^^ applied, is contained in Springer, 

Herlihy and -ffegSs (1965). 
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WELL DEFINED, CERTAIN, MULTIDIMENSIONAL PROBLEMS (BRANCH [5]). There are 
two approaches to this situation. First, one might try to express all the 
measures- in terms of some common measure*, such as dollars. Suppose, for 
example, one must decide whether or not to install an expensive computer 
system to individually manage student instruction, and that test^ have 
Shown a decrease in the average length of training time under the comput- 
erized system. There are multiple measures in this decision, dollar cost 
and average^ training time. These could, however, be reduced to cf single 
measure, dollars, if one could express the worth, in dollars, of having 
a student complete training earlier. There are a number of other techniques, 
some fair y simplistic and some. quite elegant, for reducing multiple criteria 
to a single criterion. Thp«;p will ho H-icr..,cc^^ ^...u,4. x„„a.u-.. . . 



Single criterion. These will be discussed somewhat further in section C, 
r";:r- ,^^*\^|"^"''"g preferences for Outcome". Once a single criterion is es- 
solv th bl simulation or optimization techniques can be used to' 



Second, in some problems it is easier to work directly with multiple 
criteria^ than to try to reduce them to a single criterioh. If the problem 
IS solved, by a simulation technique, this is no particular problem; outcomes 
projected by the simulation technique are characterized by many measures of 

^mnn^1hT'^^'*^^^^^'" decision maker must choose 

among them Thus, although he may be able to avoid an explicit specification 
°; offs he IS wilTing to make among objectives, he cannot avoid 

striking some kind of balance among them in the final analysis. If one 
wishes to use an optimization techniqi^ to solve the problem then working 
with multiple criteria -hec^omes very difficult. It is possible to-make some 
use of optimization; for example, x^e might do a number of optimizati6n cal- 
culations, each time using a different measur^e of effectiveness as the 
optimized criterion. Witn some luck, this would "^narrow^the choices down 

effectively solve the problem. In the more general^case, though. 
It IS difficult to make good^use of optimization. ? ^ - ^ 

WELL DEFINED, UNCERTAIN, UNIDIMENSIONAL PROBLEMS (BRANCH [6]). This has ' 
been an extremely important special case, particularly in the financial 
literature, and has given rise to s number of ideas, all of which are de- 
signed to translate this problem to an equivalent problem under certainty, 
so that the usual simulation and optimization solution techniques can be 
used. Before discussing these approaches, a bit of terminology must be 
introduced. In an uncertain problem one does not, by definition, know in 
advance the exact value of the outcome for any decision. One always knows, 
though, that for a given decision some results are more likely than others 
It is possible to express this knowledge by describing, or assessing, a 
probability distribution for the outcome. Given the probability distri- 
bution one can calculate a number of data, the most important of which are 
the mean or expected value a measure of the average v&lue of the outcome) 
and the standard deviation (a measure of , the spread, or variability, in 
the outcome). It is also possible to calculate the chances of any given 
value of the output being exceeded. 
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The first approach to sol ving ^these problem's ignores uncertainty as 
long as it stays within predefined 'limits. In setting up the problem, the- 
decision maker can specify such cpnstrain,ts as his maximum allowable loss, 
or the probability that costs, for example, exceed a given figure. Within 
these bounds he uses some simple measure of outcome, usually the expected 
^value, to solve t.is problem. In this way he translates the uncertain 
problem into a constrained problem under certainty, which is readily 
solved by certain optimization techniques. 

A second approach does not utilize the^probabil ity distribution as such 
but embodies such ideas as "Let's assume that the .worst (or best) possible 
event will occur, then maximize our gain under that assumption." It has 
been pointed out that this approach can lead to overly pessimistic (or 
optimistic) decisions/ so a variation has been developed vyhich allows the 
.decision maker to express his personal attitudes toward risk by picking a 
value for a "pessimism factor," which is then used to balance the best and 
worst cases. Still another variation assumes that all uncertain events are 
equally likely, then maximizes the expected value of the criterion. ' 

The third approach consists of defining a new, certain measure of effec 
tiveness, most commonly by subtracting a constant times the, standard devia- 
tion from the expected value of the outcome.. The rationale is that, varia- 
tions being equal, one would choose the alternative with the higher expected 
value. If variations are not equal, then one must have a higher' expefcted 
value to offset the additional risk of the larger variation. One is allowed 
to set the degree of offsetting required by picking the value of the 
constant referred tb above.. 

' •* 

All these approache5->are useful in certain circumstances, but can be 
shown to lead to Irrational decisions in oth.er cases. A more general ap- 
proach, known as utility th^eory, can be shown to be valid for all problems, 
given that one believes some basic assumptions feibout the meaning of the 
term "rationality." In the utility approach, the decision maker expresses 
his attitudes toward risk in the form of a curve, called a utility curve. 
The utility curve is then used along with the 'probability distribution 
mentioned above to calculate a m&asure of "goodness"'' which accounts auto- 
matically for ,the uncertainty in the results. This utility measure can 
then be used,vwi/th any of the techniques discussed under branch [3] to solve 
the problem"; 'This approach, although unfamiliar to many managers, , is 
straightforward. 

fclL DEFINED, UNCERTAIN, MULTIDIMENSIONAL PROBLEMS (BRANCH [7]). Sometimes 
^the^snteria can all be expressed in terms of a common measure j ais wa5 
discus^^for problems with multiple criteria with no uncertainty (Branch 
[5]). ToNUke the same example, it might be possible to approximate the 
•worth, in dollars, of shortening training by one day, then express the un- 
certainty bothMn the cost of the computer system aj^d in the number of days 
by which trainin^Nwould be shortened in terms of a single, uncertain, total 
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brlnch llf^^' '^^^ P^°^'^^^ could now be solved by methods discussed under 

■ _ In many problems it is impossible to express the criteria in terms of 
a single criterion. Another approach exploits the ideas' of utility theory 
discussed briefly above. In the case where a single criterion exists in an 
uncertain problem, one expresses his attitudes toward risk in the form of a 
utility curve. In this case, where multiple criteria exist, one can, in 
theory, do the same thing,, except that the utility curve becomes' a 3- 
dimensional or higher dimensional curve, a utility hyperplane. Because of 
practical difficulties, it is possible to determine what this hyperplane 
looks like only for certain special cases. Fortunately most real problems 
can be treated as one of the special cases, so the multidimensional utility 
approach can be an extremely useful analytical tool. Even if a problem is 
one which doesn't fit the "special case" category, one can usually net a 
good approximate solution by treating it as if it were, then seeing how 
sensitive the results are to the utility assumptions. 

. A third approach to this class of problems consists of using heuristic 
approximation techniques, or in more everyday language, using reasonable 
Ideas that seem more-or-less to work. One might, for example, select what 
he considers the most important outcome measure, get a rough idea of the 
probability distribution of that outcome, then make a tentative decision 
based on that; he would then check the other outcome measures to make sure 
that his solution wasn't ridiculous before making the decision final. 
Another commonly used approach has been to set "aspiration levels" on all 
the criteria, then' to search for a decision alternative which has a reason- 
able chance of attaining all the aspiration levels. In choosing training 
methods, for example, one may set limits on the cost and the length >of 
training, the physical facilities needed, and the prerequisites pn tl.a 
student input, then choose the method which seems to have the best chance 

"'^^^^"S^the limits. The major problem with such heuris-tic methods is 
that they depend heavily on the .ingenuity and judgment of the human ' 
decision maker, and humans ca-n be shown to be notoriously poor processors 
Of uncertain, multidimensional information. Both the methods mentioned 
above, plus many others one could conceive of , can lead to bad decisions at 
times. Somewhat more structured approaches can usefully supplement, though 
not supplant, the capacities of human judgment. , i y. 

. ILL DEFINED PROBLEMS (BRANCH [8]). Solving an ill defined problem requires 
tirst shaping it into a well defined probl-em; doing this requires a dif- 
ferent type of effort than discussed so far. Any real situation is infi- 
nitely complex in detail, so the first step in analysis is to identify the 
major issues and constraints in a problem. Next, the major action alter- 
natives should be outlined, and some thought given to the impact of each 
possible acftion upon the major "issues . Thtrd, the most promising alter- 
natives ^re selected for further study, then thi^ process is repeated. 
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Thus, the process of analysis is cyclical rather than linear. With each 
cycle the problem and the issues become successively better defined until 
the problem can finally be effectively defined and solved. In the early 
stages of problem definition formal approaches are of limited use, as the 
problem 'is mainly one of encoding impressions, knowledge, and attitudes. 
Even at this stage, however, systematic methods of thinking can pay divi- 
dends. A systematic approach to decision making, both in ill defined and 
well defined situations, is outlined in the next section. 

HOLE OF COMPUTER-BASED METHODS 

Computers have the capacity of processing enormous amounts of data at 
staggering speeds with excellent accuracy. In addition, it is possible to 
build a good deal of sophistication into computer programs. This can, 
under the proper circumstances, allow a manager to supplement his own ex- 
pertise with the intelligence which went into designing the computer pro- 
gram. For these and other reasons piles of computer printouts, and occa- 
sionally computer time sharing terminals, are* becoming increasingly common 
sights in managers' offices. Ihe precise role played by the computer in 
the decision making process can, however, vary a good deal. One critical 
dimension to the computer's role, namely the relative involvement of the 
computer versus the manager in making decisions can be depictedas shown 
in figure 2. ' 

At one extreme, the computer can be used simply as a^'data gathering 
and summarizing tool. The manager retains not only all decision making 
prerogatives, but the bulk of the analytical workload. The computer may do 
a minimal amount of computing (summarizing, calculating ratios, computing 
variances, etc.), but the primary use of the computer in this mode is to 
feed data to managers for their analysis and interpretation, and to handle^ 
routine paperwork. The bulk of current applications is of this type. 
Managerial and cost accounting systems, payroll systems, and many produc- 
tion control systems, to name a few examples, can be classified in this 
category. 

The prevalence of - such data gathering and file mafntenance systems 
has led to the creation of 'sizeable and reasonably complete data bases. 
These data bases are, in most cases, potentially valuable for a variety of 
managerial purposes. Data gathered and stored as p^art of a payroll system, 
for example^, is frequently useful in the analysis of productivity and the 
projection of employee expenses. Data from a billing and invoicing system 
can be used in inventory, control . The Navy maintains extensive records 
on equipment maintenance and failure histories as part of the 3-M system; 
these data are potentially useful to the training command in indicating 
problem areas and possible training deficiencies. The usefulness of a 
given data base for a new purpose is limited, however, by the organization 
and scope of the data.. Usefulness of 3-M data to the training command, for 
example, is limited by the fact that the training record of the man respon- 
sible for given equipment is not recorded. 
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MANAGER MAXIMALLY INVOLVED, 
COMPUTER MINIMALLY INVOLVED 



MANAGERS MINIMALLY INVOLVED, 
COMPUTER MAXIMALLY INVOLVED 



> DATA GATHERING AND SUMMARIZING 



V SIMULATION 



> OPTIMIZATION 



> CLOSED LOOP SYSTEMS 



FIGURE 2. INVOLVEMENT OF THE COt^PUTER VS THE MANAGER IN DECISION MAKING 
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A more complex category of applications involves using the computer 
to analyze data through simulation models. As explained earlier, simula- 
tion techniques involve representing a system in a mathematical form.^ 
Consider, for example, a simulation model to predict student thr^dughput 
in a training program. There are a number of factors which clearly affect 
throughput, such as the nature of the training to be performed and the 
skill level desired, instructor availability and skill, the amount and 
type of training equipment available, student intelligence and motivation, 
and so on. The builder of the simulation model attempts to determine which 
of these factors are most important in determining throughput, and just 
how the criticaj factors interrelate to determine the throughput. .He then - 
puts these relationships into a mathematical form which can be used in dT" 
computei^ized model. Once this is done, the manager can use the simulation 
model in a number of ways--to forecast some figure of interest, such as 
the budget required toproduce^a given student throughput, to perform 
"what "if" analyses, to plan and design a new system or a change to the 
present, system, or in many other ways. A feature shared by all simulation 
models is that the human, the manager, remains in the decision loop. The 
computer singly projects the effects of a set of as^gumptions , and it is up 
to the manager to examine the results and either change the assumptions 
'and perform additional analysis, or to make a final decision. 

The..next level of computer application involves ^the use of computerized 
optimization techniques. Optimization techniques, being designed to auto- 
matically determine the best possible decisions in a given situation, have 
the capability of removing the manager from direct involvement in the 
decision process. Usually, however, the manager is involved in interpreting 
and implementing the analytical results. Most optimization techniques are • 
designed to yield information not only about the optimal decision, but 
'information on the sensitivity of the result to various data and a'ssumptiops 
used in the model. The purpose of this is to allow the manager to estimate 
the impact of factors not included in the model , and to explore the f^easi- 
bility of alternatives other than those derived by the optimization technique 
Thus^ the manager and the com'puter usually supplement one another in the 
decision process. 

Finally, there^are some cases when it has been shown useful to el*imi- 
nate the manager ..from the decision process altogether.* In ma/iy inventory 
systems, a computer not pnly sets reorder points and quantities, but places 
the order as well. The only impact the manager has on this process is to 
occasionally review the system performance and to adjust the rules by"which 
the computer makes its decisions. Production scheduling and process control 
are other areas where this "closed loop" approach^has been applied. All 
these applications occur in situations which are 'repetitive, and where the 
major variables in the decision process are known. "v. 
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C. OUTLINE OF FORMAL DECISION ANALYSIS 



The decision aids outlined in the previous sefit^^rffTFrfchough u/eful 
do not provide a general method of attacking^jBtisFmanagerial itec^ ' 
problems. In these other areas, nearlv^+Tdecisions are made intuitively, 
and there are some circumstances wherfsome better method than intuition 
IS desired. Perhaps the decisioffis of major importance, or the com- 
• plexities and uncertainties of/the problem are such that tfiere is a need 
to integrate the expertise anoT knowledge of a number of people in the 
organizationj on perhaps therA ts a need to explain the issues and trade 
otts in the problem to someoneVelse, either a superior or an ally. In any 
event, there ar^ circumstances Wn a rational, systematic method of 
jutlining theydecision process is^ieed^d. The bag of techniques for doing 
th^_is>frowrr collectively as ."decision^lysis" or sometimes "decision ' ' 
theory.' Very briefly, use of these technftiu^s>I4ews one to:' 

1. Outline, all alternatives and to consider all possible consequences 
of each alternative iri,a systematic way, ' 

2. Break a^.large, colnplex problem down into a series of smaller, 
simpler problems so'that different experts or organizational units can 
contribute to the solution ofjjthe problem in their particular areas, • 

3'. Specify dfid quantify uncertainty, and determine how critical the 
uncertain variables are, ' 

4. Specify, in a logical manner, thetrade offs one is willing to 
make among outcomes, ' . 

.5. Determine the worth of gathering further information, and finally 
a measure of how jnuch better that decision is than any other alternative 
This last point is useful in deciding, for example, whether ^^actors ignored 
»in tne formal part of the analys-is could possibly change the- decision. 

Basically there are four steps to decision analysis: (l) structuring 
the problem, (2) -determining uncertainty, (3) 'determining preferences for 
outcomes, and (4) obtaining results. 

STRUCTURING THE PROBLEM 

Consider, for the purpose of illustration, a highly simplified decision 
problem. Suppose you are trying to decide whether to invest $1000 of your 
money in a bank, where you are assured a 6 percent annual return, or in a 
business venture. If the business is successful, you would receive a 100 
percent return over the next year, while if it went broke you would lose 
everything. You will cash in your investment at the end of one year in 
any case. To keep the problem simple, assume these are the only two 
possibilities, and that there are no other uses of your money that you wish 
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to consider. Finally, .suppose it is- possible to buy some information con- 
cerning your problem, A business expert of your acquaintance has heard of 
two other business ventures of the type you are interested in; for a $100 
.fee he Would research the problem for you to find out if tha two were 
successful. This problem can be diagrammed in the form of a decision tree, 
as showh in figure 3. Note vthat the decision t»;ee lays out, in chrono- : 
logical order, all possible decisions and uncertain events; ty convention, 
the decision points are represented by squares and the uncertain events 
by cincles. If you invest in the bank, then the decision tree shows a sure 
re-tarn qf $1060 (the original $1000 plus the 6 percent interest) at the 
end of ^the /ear. Similarly if you invest in the business you will have 
either $2000 or nothing at the end of the year. If you purchase the informa 
tion4 then you will find out that none, one, or two of the other ventures 
were successful . This is treated as an uncertain event since you do' not 
know, in advance which of these is the true case. After receiving this 
information, you can decide on the bank or the business investment, with 
thj^ possibilities of gain pr loss, as shown. Note that the $100 cost of 
t^e, information has been taken into account in calculating the payoffs. 

Even'i'or this simple' problem, the decision tree is an extremely useful 
^device for organizing one's thinking, for decomposing a Targe problem into 
/a series of smaller ones, ^tnd foir gathering information. A decision tree 
also serv'-ys as a good conrnqni cation tool in outlining a problem to some- 
one else. Decision trees have xhe advantage of , being easy to draw and 
manipulate, making them useful even when high-powered analysis is not 
warranted. ; ' - 

DETERMINING UNCERTAINTY ^ 

A critical, factor in the investment problem outlined above is the . 
probability of the business being successful. One rarely knows in advance, 
of course, what the odds are, but usually there is at least a vague , • 
impression of some kind (the business "looks promising" or perhaps "seems 
risky"). It is possible to quantify these impressions by interviewing the 
decision maker, or better yet, an expert in the field, to determine a 
probability distribution of the odds. The probabiVity distribution can 
then be used to determine the'attractivehess of the business venture. The 
probability data can also be 'combined with objective data, such as is 
obtained on the "purchase information" option, to determine whether th 
venture still looks good after the information comes in, and. to determ 
whether or not the information is worth the $100 cost. 

DETERMINING PREFERENCES FOR OUTCOMES 

In our example a single measure of effectiveness, namely the. amount 
of money at the end 'of the first year, is used. Even in this case, it is 
not clear. how to proceed; for one thing, people's attitudes towards risks 
differ. It jnay .well be optimal, for example, for a poor -man to pass up an 
otherwise attractive investment because the chances of loss are too great. 
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Even when two individuals have the same wealth, one person may be more 
willing to take chances than the other. A method for dealing with these 
considerations was first suggested by von Neumann and Morganstern (1944). 
Their idea was to pick a "best" outcome and a "worst" outcome ^which are 
at least as good and bad, respectively, as any outcome you expect to get. 
In the investment example, the best and worst possible outcomes are $2000 
and -$100. Then for each possible intermediate outcome , one must assess 
a probability such that the intermediate outcome is" exactly as attractive, 
as a gamble between the best and worst outcomes. This probability is 
called the "utility" of the outcome. For example, consider the $1060 
which we would receive by investing in the bank.* We assess the utility 
of $1060 by determining a probability, which we will call ^p, so that the ^ 
gamble in figure 4 is neither more nor less attractive than $1060 for sure. 
If p we're nearly one then 'the gamble would be more attractive, while if p . 
were nearly zero the $1060 would be more attractive, indicating that there 
must be some* value of p between zero and one where the choices are equally 
attractive. Note tha^t the value of p chosen would vary from individual to 
individual, depending cm the dehisioh maker's personal attitudes toward 
risk. \. 

Finally, von Neumann \nd Morganstern (1944) point out that the value 
of p (or the utility) is a measure of the relative attractiveness of the 
$1060 consequence, and proveXthat the expected value of the utilities of 
end points is a valid decisio\i criterion under uncertainty. To solve the 
investment problem, then it is necessary only to assess the utility for 
each end point, calculate the expected utility for each decision, which, is 
easily done, then choose the action with the highest utility. 

The validity of the utility approach depends upon certain behavioral 
''axiGms, or observations on rational behavior. The major ones are: 

1. Given two consequences A and B, then either A is preferred to B, 
B is preferred to A, or both A and B are equally attractive; 

c 

2. If A. is preferred to. (indifferent to) B and B is preferred to 
(indifferent to) consequence C, then A is preferred to (indifferent to) C; 

' 3. Given a utility assessment problem such as is outlined in figure 4, 
it is always possible to find a value p such that the gamble and the "for 
S'.ire"' amount are equally attractive; 

4. If consequence A is, preferred to consequence B then of two dif- • 
ferent gambles between A and B, the one- offering the larger chance at A 
is preferred. 

These seem to be reasonable assumptions, but it has been observed that 
people do not always act according to the axioms. This fact indicates. that 
utility theory may not be a good descriptive theory, but emphasizes its 
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YOU MAY HAVE EITHER 

(§) lljOoO FOR SURE 




-$100 



WHICH GIVES A PAYOFF OF $2000 
WITH PROBABILITY p AND A PAYOFF 
OF -$100- WITH PROBABILITY (1-p) 



YOU MUST SET THE VALUE OF p SO THAT ® AND (b) ARE EQUALLY ATTRACTIVE 



- FIGURE 4. ■ UTILITY ASSESSMENT EXAMPLE 
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potential usefulness as a prescriptive theory; i.e., one which indiGqites 
improved decision methods. 

In the case where multiple measures of effectiveness must be used, 
the same basic ideas cf utility theory hold. Due to practical problems, 
however, it "is possible to assess a multidimensional uti'^ity function only 
if certain assumptions about one's -preferences hold. For most problems 
the assumptions are valid, making it possible to use the utility approach. 

OBTAINING RESULTS 

. * Once the problem has been structured and the probabilities and pref- 
erences assessed, obtainin§!-K^esul ts is a straightforward computational 
matter. Simple problems can be.^sol Ved by hand, while for more complex 
analyses a number of computer methods can be used. The hardest part, by 
far, in a decision analysis is in structuring the problem and gathering 
data, rather, than in calculating the solution. A good deal of "art" is 
required in selecting the portions of a problem to explicitly represent, 
since a decision tree can rapidly become overly complex if too much detail 
i§ included. Analysis, is generally a cyclical, rather than a linear 
process. This comment applies particularly to this type of analysis. The 
first cut at a problem should outline major alternatives and outcomes only 
roughly. After "lopping-off" the least promising branches of the initial 
tree, the remaining part can be elaborated, and this process continued as 
long as is necessary. 

D. USE OF FORMAL DECISION THEORY IH ORGANIZATIONS 

In section II. A some questions were raised xoncerri-ing the role of* 
formal analysis in real organizations, indicating tha.t it is important to 
identify the types of problems for which formal techniques would be i:seful. 
One way of approaching this. issue is to summarize successful applications 
of the method. . ' ^ } 

APPLICATIONS OF DECISION ANALYSIS . * 

Decision analysis, in its present form, resulted from a marriage 
between a particular school of thought concerning statistical analysis 
(the so-called "Bayesian" approach) and the von Neumann-Morganstern theory 
-of utilities. Von Neumann and Morganstern were interested primarily 
' v^in applications in economics, giving that side of the union a strongly 
" ^business oriented bent, and as the union was consummated largely in 
^graduate schools of business, most of the applications work has been in a 
business setting^ Reinforcing this propensity has been the fact ^hat 
■business problems have a natural, easily measured, common measure of 
effectiveness, namely profit. 
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BUSINESS APPLICATIONS. An important early application of decision theory 
is Markowitz (195.9). Markowi^tz Was concerned with the rational invest- 
ment of funds in a portfolio of potential investments, each characterized 
. by an expected monetary return ajnd a de^ee of riskiness. The 'portfolio 
problem is how to determine the investment, or mix of investments, which 
optimizes the^return/risk trade off. • . 

The portfolio selection process has a direct analogue- in business 
decisions involving capital investment. Matheson (1969) discusses an 
analysis of new pi oduct development alternatives. Briefly, a major manu- 
facturing research company has developed two compounds for a particular 
market, and the decision must be made to do final development o^n neither, 
.- both, or only one of them, and if the latter, to decide which one.- 

Cook (1968) presents another analysis of a product development, this 
time, in the atomic power field. Four different product development 
alternatives for atomic electric generating stations were analyzed. A 
complex computer model of the market, .costs, demands, and sales aspects of 
the problem was constructed and evaluated by decision analytic techniques. 

Additional analyses of a similar nature are described by Laessig & 
Silverman (1974) and" by Frederick (1973) who describes a product pricing 
problem for butadiene, a petrochemical product, in which multiple criteria 
were considered. Huber (1974) reviewed a number of field studies, primarily 
business oriented, in which multidimensional .utility models were used. A 
more complete and general discission of risk analysis in capital projects 
is contained in Spetzler (1968). Spetzler interviewed a number of execu- 
. tives of a company, then used the resultant utility assessments to formu- 
late a .corporate risk policy. 

Decision analysis has also been shown to be useful in settings other 
than investment problems. Keeney (1969) discusses an application of multi- 
dimensional utility theory to determine the optimal organization of a 
telephone network. The object was to maximize the degree of service, as 
determined by the percentage of time the lines are available to two dif- 
ferent customer groups. 

In Brown (1970) "the results of a survey among firms using, or who have 
used, decision analysis are reported. The firms surveyed included organiza- 
tions with several years of active -experience in decision analysis, some 
where the method is fairly new but is in active use, some where there is 
interest, but little application, one or two where decision analysis has 
been a disappointment, and two consulting' firms with expertise in the area. 
Brown found that general decision making procedures are not radically 
dffected by the presence of decision analysis, but that individual decisions 
are often profoundly affected to the good. The consensus among the survey- 
participants was that the methods had enormous potential which is not yet 
realized. Major problems- seen in using the method are: (1) management 
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education, (2) communications between the analysts and the managers for 
whom the analysis is done, (3) the difficulty in many organizations in 
identifying who is responsible for specific decisions, and (4) organiza- 
tional obstacles. Brown concludes that "If there is one- dominant feature 
that distinguishes the successful from the less successful applications of 
DTA (Decision Theory Analysis), judging from the findings of this survey, 
it is the organizational arrangements. . . . The most successful appears 
to be the 'vest pocket' approach, where the analyst works intimately with 
the executive and typically • reports to him."..," 

MEDICAL APPLICATIONS. A rich literature has grown up describing applications 
of decision theory to medical problems. Among the reasons for this are: 
medical decisions have important consequences in cost, suffering, and 
death; medical problems are complex and involve uncertainty; the volume 
and fragmentation of knowledge requires an effective integrating structure; 
data are widely available and relatively easily obtainable; and public 
interest in medicine is high. Although medical decisions per se are not of 
interest in this research, the methodology of applying decision analysis 
which is 'demonstrated in this field is. In particular, medical decisions 
typically require consideration of multiple objective criteria, and a 
number of techniques for dealing with this problem are described in this 
literature. 

A number of articles describe the application of decision theory to 
specific medical diagnostic or treatment problems. Giauque and Peebles 
(1974) discuss analysis of the treatment of strep throat and. rheumatic 
fever, developing in the process a scheme for evaluating consequences 
with as many as 10 attributes. Ginsberg (1971) performs a similar analysis 
for the pleural-effusion syndrome (which involves fluid in the lung cavity), 
while Ginsberg and Offensend (1968) discuss a*diagnostic problem in spinal 
bone disease. The approach to the multidimensional consequence evaluation 
in both these cases was somewhat simpler. Thomas et al (1973) analyze the 
diagnosis of heart disease, while Schwartz et aX (Tg^TT discuss hypertension 
(high blood pressure). Some of the papers tipsirticularly Giauque and Peebles 
(1974), Ginsberg (1971), and Schwartz et^ ai (1973)] contain general dis- 
cussions of decision analysis in addition to the specific studies. 

More general approaches to broad problems, are contained in Giauque' 
(not yet published) and Lusted (1971). Giauque discusses a utility approach 
to measuring the quality of health care, with a particular application in 
the treatment of hypertension. Lusted discusses the use of decision theory 
in interpreting X-rays. Lusted (1968) contains an extensive bibliography 
of other medical analyses. 

PUBLIC SECTOR APPLICATIONS. Public sector applications are partjcuja'rly 
difficult to analyze since they have neither the natural measurement/ 
criterion. of profit found in business applications nor the data availability 
of medical applications. Decision analysis offers methods both for dealing 
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with the multiple criteria required in public sector .analysis and the 
uncertainties^ caused by lack of data, leading, to a significant volume of 
literature on public sector applications." 

*• Howard, Matheson, and North. (1972) discuss the problem of deciding 
whether or not to seed hurricanes with silver iodide. Experiments with 
seeding have shown promising results, but a decision to seed a hurricane 
bearing down on populated areas carries legal and moral consequences. 
Howard et ail_ used decision analysis to examine the problem and to explore 
other decision, alternatives besides the "seed" and "don't seed" alterna- 
tives. In Giauque (not yet published - II) a scheme to determine an optimal 
method of oil spil'^ cleanup in harbors, depending on the geographic and 
climatic conditions at the harbor, is presented. Keeney (1969) explores 
blood bank inventory control and cost/benefit relationships of depth sur- 
veying in the Cape Cod Ship Canal through use of a multidimensional utility 
analysis. 

A second group of papers are broader in scope, but still deal with 
well defined problems and priorities. Gear (1974) and Roche (1972) present 
analyses of planning in education. Gear, after discussing approaches to a ' 
number of common educational decision problems, presents an analysis of 
secondary school pupil allocations between adjacent geographical areas. 
Roche discusses an extensive' investigation- into the problem of resource 
allocation among different subject areas in a secondary school. This 
involved determining the trade offs the school' board and school administra- 
tion are willing to make among proficiency levels in the various subjects. 
. Other application areas include space and military planning. Matheson 
(1969) presents a method for planning payloads on unmanned Martian explora- 
tion vehicles. Power (1973) discusses an interactive system, utilizing 
decision analytic concepts, to plan cost and schedule estimates for anti- 
ballistic missile programs. ' . 

Stanford Research' Institute (1968) conducted a study for the Mexican 
Government, in which a strategy for electrical power system expansion for 
the entire country over the next 30 years was derived. This involved fore- 
casting a complex array of power needs, technical advancements, price move- 
ments of various fuels, and so forth, over this time frame'. In addition, 
a number of social trade offs had to be considered. For. example, the im-- 
pact on employment, self-sufficiency, side benefits, and technical exper- 
tise required are very different for say, nuclear versus hydroelectric 
generating plants. This study is referred to and discussed in Matheson 
(1969) and Howard (1971). 

In a separate study, Keeney and Nair (1974) discuss the complex issues 
and trade offs involved in licensing nuclear power plants within the United 
States, and propose a decision analytic based approach to solving these 
issues. Hammond (1971) and Ellis & Keeney (1972) derive methods to analyze 
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problems of strategic military planning and air pollution control, resDec- 
tively. Finally, deNeufville and Keeney (1972) consider the possibilities . 
for future development of the Mexico City, .Mexico, airport. A number of 
effectiveness measures were used, specifically, noise problems, cost, capa- 
city, safety, transportation time, and the number of people displaced bv 
the airport expansion. h y 

APPLICABILITY OF DECISION ANALYSIS 

The overriding impression one gets from reading the applications litera- 
ture IS the rmportance of the relationship between the analyst and the 
decision maker. ? What is studied does not seem to be nearl'y ac critical as 
how the study is performed. In the words of Keeney and Raiffa (1972) 
The metadecision of v^hether or not to do formal analysis cannot be divorced 
from the questions of organizational structure, of the" personal incentives 
for the people involved, and of the quality of the analysts." Brown (1970) 
also emphasizes the quality of the analyst-client relationship. The above 
amounts to saying "Get a good analyst and a motivated manager, net them 
working well together, and no matter what the problem is you'lTget a 
good analysis." This is the same thing that operations researchers have been 
saying for years, and it seems to be true in decision analysis as well. 

Some additional light on this issue can be obtained by referencing 
figure 1. Branches of the taxonomy tree marked with an asterisk are those 
where decision analysis is most likely to be useful. If problems are well 
defined, certain, and have a single decision criterion (measure of effec- 
tiveness), the particular strengths of decision analysis are not really 
called into play. In other types of problems, though, the usefulness of the 
method can be dramatic, either alone or in. conjunction with othe> techniques 
The structural aspects, of decision analysis are helpful i;i defining problems, 
specifying the magnitude of uncertainty, providing for contingent decisions, 
and determining the sensitivity of results to assessments and assumptions. 
The utility formulation allows one to specify objective criteria valid under 
risk, and to reduce multiple criteria to a single criterion. 
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SECTION III 

DECISION MAKING IN THE NAVAL EDUCATION AND TRAINING COMMAND 

In the previous section a number. of general observations concerning 
decision making in large organizations were summarized. With this back- ^ 
ground, decision making in the NAVEDTRACOM is now discussed. Source mate- 
rial for this section was gathered both during field interviews with NAVED- ^ 
TRACOM staff personnel and from written material. A particularly useful 
discussion of- the functions performed by the NAVEDTRACOM, including speci- 
fically the decision-making functions, is found in TAEG Report 12-1. . 

A. HISTORICAL AND ORGANIZATIONAL SUMMARY OF THE NAVEDTRACOM 

The NAVEDTRACOM was organized in August 1971. Prior to that time the 
training function in the Navy was allocated among a number , of agencies. Due 
to the difficulties of coordinating efforts it was decided to consolidlTEe 
the responsibility for all naval training-sunder the- newly formed office of 
the CNET. The CNET and hi^ staff, located in Pensacola* Florida, have re- 
sponsibility for. all naval education and training programs, except medical 
training and some training done in the. Fleet (source: TAEG Report 12-1, 
p. III-7).. Included in his duties are the responsibilities for organizing, 
staffing, and running the training function, submitting and administering 
training budgets, determining optimal training methods given the budgeting 
and congressional constraints faced by the Navy, advising the remainder of 
the naval organizations on training matters, and other related duties. The 
position of CNET. is currently filled by a Vice Admiral. 

An organization chart for the major divisions of the NAVEDTRACOM is 
given in figure 5. Five major entities report to the CNET. The CNETS is 
tasked with providing to CNET and other assigned claimants support in the 
areas of, first, material evaluation, development, production, support, and 
.validation; and, second, educational and library services, general military 
training, and international logistics. The CNTECHTRA is assigned the re- 
sponsibility of administering, staffing, supervising, and. running the Navy^s 
technical training schools. The Chief of Naval Air Training (CNATRA) per-, 
forms the same functions for. Schools involved with air training. The final 
two organizational entities repotrting to CNET are the Commanders of Training 
in the Atlantic and Pacific Fleets^ (COMTRALANT and COMTRAPAC). These 
organizations are tasked to work wi^h CNET and their respective fleet com- 
manders to implement and supervise fl^t training. 

Currently over 50 percent of all training dollars are spent on recruit 
and advanced enlisted training, falling under CNTECHTRA, COMTRALANT, and 
COMTRAPAC, and an additional 25 percent on tlight training, falling under 
CNATRA. ^ . \ 
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FIGURE 5. NAVEDTRACOM ORGANIZATION 
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B. MAJOR DECISION MAKING FUNCTIONS OF THE NAVEDTRACOM 

■ The responsibilities of the NAVEDTRACOM can-be classifie^d into 10 



major areas. These are: 


1. 


develop training requirements, 


' "2. 


analyze and plan training, , • 


3. 


coordinat°e and controT training, 

/ " , ^. 


4. 


implement trailing, 


5. 


evaluate, performance, \' 


6. 


manage resources. 


7. 


perform research, 


8. 


support training, , 


9. 


administration and internal control, and 


.10. 


miscellaneous functions. 


DEVELOP TRAINING REQUIREMENTS 



New training needs arise from two types of situations. First, as the 
hardware and systems used by the Navy change there is a need to modify the - 
corresponding training packages for incoming personnel and, in most cases, 
to retrain current operating personnel. Second, the , requirements for ongoing 
training are modified due to changes in such things, as personnel requirements, 
characteristics, input rates, etc.; changes in operational demands; or changes 
in policy. In general, a training need is defined as a need for a particular 
number of ' peVsonnel with a given skill. The need requirement first goes to 
BUPERS, where an inventory of Naval personnel with given skills is maintained, 
to determine whether the requirement can be satisfied out of, the existing skill 
inventory. If the requisite number of qualified peopje is lacking, then the 
CNET is requested (or required) to provide training to meet the need. 

As can be seen, tf^aining requirements originate outside the NAVEDTRACOM, 
as it is up to the operating Navy to determine needs and 'the responsibility 
of the training system to fill those needs. Training needs can come to 
NAVEDTRACOM via four major interfaces: 
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O" \ . ■ 

1. the Chief of Naval Operations, 

the Naval Material Conmand, ■ . 

3. the Bureau of Naval Personnel , and/or 

4. the Fleet Commands. ' ' 

L-vpn fn^PMpf " '"^"'^^ ^^^1"^^' however, in that requirements 

given to CNET are usually ambiguous TAEG Report 12-1, op. cit In addi 
tion certain types of changes, 'particularly policy changes ma^ affect Jhl" 
requirement for training greatly, but the changed^equ??emen?s marneJeJ 
be formally submitted to CNET in the fom of a training task reSremen?- 
in such cases CNET generates the requirements internally An eSole Sf' ' 
such a policy change is the decision to implement the al voluS forJe 
fn since the detemination of requirements is crit'^Il 

to the effective functioning of CNET, this function is handled as if it 
were an integral part of the training system. ^ 

DESCRIPTION OF EXISTING SYSTEM^ . The problems of developing traininq re- 
quirements can be seen to be both important and ill defined The firs? 
■p oSleS '° n^Z'?Nj;^ defined problem is to make it into a-wel?'de' eV 
Jear CNET s SrpS wIth'^'"- j!^^^'' ^"^'"Plished as follows. Each 
rffn..^ 1 t " " load" Of continuing training requirements- 

an °? ?o eT nto't '^h'h° T'V ''''''''' are routinely e^tlSe ' ' 
■ ?hrouah thP nMcf?f- New requirements,, on the othe? hand, go 

through the justification process outlined in figure 6. ' This procedure 

an^S i^^ ^i^°V' '^'^^ are^~alyzed 

ana priori ti7.ed by CNET. A request for new training generally, but not 
always, is originated outside CNET. The first, step in process ng the re- 
quest 1-s to make ar^ estima-te of the resources required to implIZt the 
request The form used to. document the estimates (CNET FoJS^ I5S0/8 see 
cia'ssrool an3"]rhn..'r^'"'''^ ''''''^ requirements and knTw s2 ^ce / 
olherpen?neni'Sa°?f°'' -^t-ructor requirements, and/ 

/ 

It is currently the case,- and it is reasonable to expect that it will 
resoi?c1s ° T^hi^'n ."f 't'^' '"''^''^ '^^^ a" es? 

ing on the particular project ^ ^ °' committees, depend- 



•1 ' " 

Refer to TAEG Report 12-1, Vol 1 (ou cit ] Pr-in^n + n-^n^- • • ^^ 
under the "Coordination and Cont ol7 t '4alys s n ' ^ann^•rla ''TnT'' 
more extensively, under the "Manage Resources" funct?ons'' ' 
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INTERNAL TO CNET 
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1 



VALID REQUEST, 
RESOURCES 
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FPGURE 6. ESTABLISHING NEW TRACING. REQUIREMENTS' 



TAEG Report No. 27 \ 
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NEW EXPANDED TRAINING PROGRAMS ^ 
CNET Form 1500 '8 {Rev, 4-74} 
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Figure 7. CNET Form 1500/8 
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Clearly, the priority af any particular program should be determined « 
by^ a nymber of factors, including usefulness to the fleet, urgency of the* 
training, length and cost of. the training, projected cost savings (if any) 
anti^cipated, etc., plus any number of such intangible but critical issues 
as who the program sponsor is, and the "political" effects of the training 
both insiide and outside the Navy. The prioritization scheme currently used 
addresses some of the issues in a formal manner and others in an informal 
manrter 0/ not at all. The formal phase consists of the use of the -scheme 
outliYie^ in ftgureJB to assign a "priority designator." This procedure is 
used for all projects except those relating to air. training; it is expected 
that the air training, command will develop a similar scheme appropriate for 
their needs. A complete description of the prioritization scheme is con- 
tained in CNET Instruction 1500.10 (11 June 1974). ' 

The factors considered in the formal scheme; and how they affect the 
priority designator are summarized below: 
- • . • 

1. The military purpose of the -train^n§^^^^ Possible ca.tegories, in 
general, order of priority, are special, str^ategic, sea control , projection 
forces, logistic support, ' and genet'al and non-warfare. Special projects 
are those which, due to their ^mpiict .on the Navy or their urgency as de- 
fined by higher authority, rtlus^'recei ve special consideration. These pro- 
jects always have the highest priority. Although strate^gic programs gen- 
erally have the highest priorities, and logistic support and general non- ^ 
Wc^rfare training the'lowest priorities, there are some exceptions to this 
rule. 



are: 



• 



Operational function served by the training. Possible categories^ 

propulsion and auxiliaries 
coimijni cations 
navigation and seamanship 

life support, including damage control and safety 

surveillance (search and detection), electronic warfare, security 
and intelligence, operational security, and cover and deception 

command and control, weapons control, and tactical data processing 

weapons (offense or defense), missile and gun systems, and specific 
primary functions of non-combatants, and 

administration, planning, and personnel . 
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F^URPO'SE OF TRAINING: 



SPECIAL 



STRATEGIC 
SEA CONTROL 
PROJECTION FORCES 
LOGISTIC SUPPORT' 
GENERAL AND NON-WARFARE 



SPECIAL PROJECTS AUTOMATICALLY ASSIGNED 
HIGHEST POSSIBLE PRIORITY 



FUNCTION OF TRAINING: 

PROPULSION AND AUXILIARY 
■ COMMUNICATIONS 
NAVIGATION AND SEAMANSHIP 
ETC. ■ 



PURPOSE/FUNCTION 
NUMERAL 



URGENCY OF TRAINING: 



NOT 



TRAINING REQUIRED BU 

AVAILABLE 
AVAILABLE COURSES NOT ABLE 

TO MEET REQUIREMENTS 
ETC. 



I 



OPERATIONAL REQUIREMENTS 
DESIGNATOR 



TYPE OF TRAINING: 

CONTRIBUTES DIRECTLY TO 
- COMBAT READINESS, ETC. 
TRAINING OF TEAMS 
ETC. 



I 



PRIORITY 
DESIGNATOR 



QUANTITY OF PERSONNEL TO 
BE TRAINED 



I 



TRAINING REQUIREMENT 
DESIGNATOR 



FIGURE S. PRIORITY DESIGNATOR ASSIGNMENT SCHEME 
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Higher priorities are associated with combat related categories (propulsion 
and auxiliaries, communications, etc.) and the lowest priority with admin- 
istration, planning, and personnel. 

3. Urgency of the training. The highest priority is assigned to 
tasks required to maintain operational effectiveness in the fleet or to 
implement CNO-directed programs. Other categories, in decreasing order 
of priority, are: 

• major training objectives to which this course contributes sig- 
nificantly cannot be met by alternate means 

• most major training objectives to which this course contributes 
significantly can be met by alternate means, but some major ob- 
jectives cannot be met completely , and ' 

• all major training objectives to which this course contributes 
significantly can be met by alternate means, but minor training 
objectives will not be accomplished. 

4. Type of training. In order of decreasing priority, the possible 
types of training are: - 

• training conducted on board the duty station or operational site 
in functions which contribute directly to combat readiness, opera- 
tional capability, or technical maintenance, 

• training of teams of personnel to perform functions within units 
of the operating forces, 

' • training which qualifies for assignment^of an existing or planned 
NEC or NOBC in units of the operating forces, or training, which 
awards certification of a skill upon successful completion of the^ 
course, 

• training which earns an NEC or NOBC of a billet ashore, or training 
in operational or technical skills similar to NEC or NOBC awarding 

• courses, or is designated as a Class C or F course, or is designated 
for achievement of journeyman level, 

• training in initial skills, or is designated as a Class A course, 
or is designated for achievement of apprentice level, or training 
for advancement in rating to pay-grade E-4, - 

• advanced training of career-development nature or designed for 
supervisory or management skills, or training for advancement in 
rating to pay-grades E-5 and above, and 

• all other types of training. 
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5. • Quantity of personnel to be trained. Categories of annual inputs 
range from over 10,000 to .less than 50 trainees per year. 

^■^,..M^I! assign a priority designator to any project, one works 

through a series of tables, reading off the categories appropriate for each 
of the Items summarized above to determine priority codes which are combined 
00 ^'^?,j3?'es to yield a priority designator, a number varying from 00 to 
J^, with lower numbers designating higher priority. 

P"ce a project has been assigned the priority designator, it comes 
before the Traimng Requirements Comnittee. This comnittee, which meets 
at least monthly, examines each proposal and delivers' a recommendation for 
.Jl nf lh"VM^?'"c^''^°^ Training Requirements Comnittee are drawn from 
eacn of the CNET Sections, from administrative levels directly below the 
Deputy Chiefs of Staff. Detailed descriptions of the- composition of the 
committee and duties of committee members are found in CNET Staff Instruc- 
tion 1500.5, 22 April 1974. 

As shown jn figure 6, the Training Requirements Committee normally 
issues one of five possible recomnendations for each project. In some cases 
the recommendation is implemented directly, and in some cases it is referred 
to a Priorities Board for further review. The Priorities Board-, consisting 
of the Deputy Chiefs of Staff of CNET, has the final authority within CNET 
on training decisions. Possible recomnendations of the Training Require- 
ments Committee, and the action required by the Priorities Board, are as 
follow (source: CNETSTAFFINST 1500.4): , a ab 

_ 1. The training requirement is considered valid and can be met within 
existing resources and training linitations. These actions are passed to 
the appropriate operations division for implementation 

u training requirement is considered valid but implementation 

should be postponed, until the budget or subsequent year. In these ca<=es a 
budget request is made, and when funding is made available the program is 
implemented. *^ ^ 

3. The training requirement is considered valid and the training pro- 
gram needs to be implemented during the current fiscal year, but resources 
or Congressional training authorization are not readily available Such 
nfiTr'^ii^^'Tl^''-^ to the Priorities Board, along with'a summary 

ot specific deficiencies, recomnended lower priority source of compensation, 
and possible recommendation for consideration at a later date. The Priori- 
ties Board then decides whether or not to implement the program and if im- 
plementation IS directed, what sources will be used for the funds. Generat- 
ing funds may require discontinuance of a lower priority project 
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4.. The trainiag request is not- considered valid and implementation 
is not recommended. In these instances the committee recommendation and 
the reasons therefore are recorded and forwarded for decision by the Prior- 
ities Board. If the Priorities Board concurs, the originator of, the require- 
ment is notified. 

5. The request cannot be processed since quantified data are lacking. 
In these cases,, the cognizant staff code is requested to .provide the missing 
data, and the request form is recycled for consideration at a subsequent 
meeting. • . 

In summary, the process of deciding which programs to implement consists 
of two major phases. The first phase, assigning a priority designator, is 
a fairly mechanical means of measuring the overall usefulness of the program 
to the Navy, while the second phase, consisting of reviews by the Training 
Requirements Committee and the Priorities Board allows for input of s one of 
the intangible considerations. 

COMMENT ON EXISTING TRAINING REQUIREMENTS SYSTEM. The general problem of 
developing training requirements, one must recognize, is ill defined. Most 
organizations deal with such situations by avoiding meaningful long range 
planning (instead, they react to crises as they arise) and by relying on 
standard rules for making decisions. Both of these modes of action basically 
ignore the existence and importance of uncertainty. The plan becomes a sub- 
stitute for reality. The process' described for developing training require- 
ments is in agreement with this observation. Ideally the role and composi- . 
tibn of the Navy over at leas^t the near future would be translated into 
lists of specific required skills, which would in turn be used, along with 
skill inventory projections, to determine training requirements. This pro- 
cess would intimately involve CNET as both the repository of training exper- 
tise and the coordinating agency for all training. To some extent this pro- 
cess is followed but only imperfectly. The operating Navy often is unsure 
of its future requirements and of the impact-of brosid level policy and force 
changes upon training requirements; thus it is unable to give the training 
command precise definitions of its needs (TAEG Report 12-1, Vol I, p. VI-340). 
Instead of actively participating with the operating Navy in designing 
training programs, the process described in the previous section has CNET 
responding to requests for specific programs. As individual training re- 
quests are submitted, it may be difficult to build an integrated and com- 
plete training program, thus the current system is rather piecemeal. The 
CNET staff, in' order to at least partially overcome this problem, does do 
some requirement definition. It must also be recognized that the above 
description understates the amount of cooperative interaction between CNET 
and the operating Navy by ignoring some of CNET^s other functions, specifi- 
cally in feedback and evaluation tasks and in training advisory roles. 
Nevertheless, it must be remembered that the ideal is continued close coope- 
ration between the training and outside commands in these matters. 
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The essence of prioritization in any organization is the determination 
of optimal trade offs airong multiple objectives. The formal phase of the 
■current prioritization scheme recognizes trade offs to some extent by making 
the priority designator depend on a number of factors— the purpose of the 
training, the function ofi the personnel to be trained, the quantity of per- 
sonnel to be trained, and\so forth. A number of other factors, however, 
such as the estimated resources needed to accomplish the -training, are 
ignored in the foniial pridritizatic^n scheme. . These are brought informally 
u !u"^"^ Requirement's Committee and Priorities Board. In addition, 
4.U 4. j^^^ trade offs buirlt into it, they are, in a 

The person who constructed the tables may have 
ntifying trade offs, but if these were more ex- 
correct the scheme for possible oversights and 
Navy's, role and needs change. The techniques 
nal utility theory are ideally suited to this, 
xplicit and logical- specification than is 
ables. There are also a number of other bene- 
a multidimensional utility approach to this 
e measures of effectiveness for training and- • 



although the current schemi 
sense, hidden in the tables! 
had a good intuition for qu 
plicit it would be easier t 
to update the tables as the 
associated with multidimensi 
task, and allow a much more 
possible strictly by use of 
fits which would result from 
problem. It would identify tl 
these 



separate these from process cbnsiderations , that is considerations of how 
the means are to be obtained. To illustrate, measures of effectiveness 
include costs, throughput, skill level attained by the trainee, etc., while 
process considerations include some aspects of the type of training given, 
location and setting of the training, training aids used, and so forth. The 
current scheme contains some elements of both purpose and process. Also, 
there are some legitimate purposes of training, such as cost reduction, and 
n ihly relevant considerations!, such as estimated resources nfeeded for the 
new program, which are not forjmally addressed by the current scheme but 
which could easily be included in a multidimensional utility formulation. 
Finally, a new scheme could yjeld a priority designator which could be 
calibrated in terms of some mksure, such as dollars. It would then be 
easy to judge the impact of adding or dropping any given project. The 
cise manner in which multidinjensional utility theory could be used to 
struct a new scheme is too technical to describe here, but the 
is moderately straightforward. 



pre- 
con- 
technique 



The final steps of the /priori tization process, review and final priori- 
tization by the Training Requirements Committee and Priorities Board, allows 
for a broad based review arid negotiation process by the managers involved 
Every successful manager h^s learned that there are some facets of any problem 
which are difficult to worK into a formal analytic structure but which are 
critical to any decision. / For example, factors involving morale or humani- 
tarian issues are difficult to quantify. In addition, the opinions and 
desires of some people . irj an organization carry more weight than those of 
others. Thus the best flomial prioritization scheme possible would fail 
unless there were some ^i (^formal mechanism for working the intangible consi- 
derations into the prior/ity ranking. The two committees allow for such an 
input, thus their existe/nce and continued functioning are critical It 
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has been repeatedly demonstrated, .however, that although the human minimis 
extremely good at some kinds of complex synthesis tasks, it is very bad at 
systematically processing complex data. In particular, humans are poor at 
consistently choosing optimal combinations of factors, especially when many 
factors must be considered simultaneously. Sinc^ prioritization requires 
this kind of thinking, it is advantageous to inc de as many factors as 
possible in a formal prioritization scheme, and t n use managerial judge- 
ment to adjust the results to allow for the factoid which were omitted. 
To do this job well, the committees require a formal scheme which (1) con- 
siders all quantifiable factors in deriving a priortty rank.ing5 and (2) 
yields not only a priority ranking, but a measure- of the strength of the 
priority differences as well, so the committees can determine "the "cost" 
of altering selected priorities. The multidimensional utility approach - 
discussed above does precisely that. 

RECOMMENDATIONS FOR MODIFYING THE TRAINING REQUIREMENTS SCHEME. Specific 
recommendations for modifying the existing training requirements system are 
as follows: 

1. As far as possible, encourage close and continual cooperation 
between CNET and the operating Navy (including the chief of naval material) 
'in defining long range training requirements. It would be useful to set ♦ 
up a permanent committee to make such projections on a regular basis. 

2. Develop a new formal prioritization scheme to allow the considera- 
tion of all quantifiable factors including resource requirements, in deter- 
mining priority scores. The technique of multidimensional utility theory 

is ideally suited to developing this scheme. 

3. -Slightly redefine the role of the Training Requirements Committee 
and the Priorities Board so as to include quantifiable factors in the formal 
prioritization scheme, rather than in the factors considered explicitly by 
the committees. 

ANALYZE AND PLAN TR./\INING 

The "analyze and plan training" function determines the what, how, 
where, when, and who of training, and yields an estimate of how much it 
would cost. The first step is to determine the performance requirement to 
be met, or the "what" of training. The performance requirement defines the^ 
job to be done by the person receiving the training and identifies the quali 
fications required to do it. The differences between required qualification 
and available qualifications then determine the specific training goals and 
tasks, which are further broken-down into specific elements to be taught. 
These elements form the basis of the training course. 
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The next step determines the "how" and "where" of training. Any given 
course can, generally, be taught by any of a variety of techniques. The 
most obvious method of presenting training is to start a new course, task 
a shore school with providing the manpower and space to teach it, develop 
an outline, provide equipment, then plan a regular schedule of presentations 
This can be a major undertaking with high costs, and sometimes considerable 
savings can be realized by alternate methods, particularly when the proposed 
training involves a relatively small number of people or when demand is 
sporadic. Often an' existing course can be modified or extended so that the 
new requirement could be attached to a related course. This may be particu- 
larly attractive if courses were constructed in a "modular" fashion, with 
modules constructed to cover single subject elements. A training require- 
ment can sornetimes be contracted out to a manufacturer or conducted on board 
a duty station- rather than in a school. Courses can be-. taught in a number 
of different locations, or students can all be brought to a central location 
Historically, training in the Navy has been dispersed among a number of loca- 
tions. Currently there is an effort underway to centralize training as much 
as possible,' to restructure training classes to support a new classification 
of enlisted. personnel , and to reallocate specific training tasks among A 
schools, B schools, and on-the-job training. Even when the decision has 
been made on how and where to teach a course, decisions are still required 
on what types of training equipment should be provided, what kinds of audio 
and visual aids will be used, when programmed instruction texts are appro- 
priate, etc. 

The final steps of the analysis and planning of training are to deter- 
mine a training schedule (the "when" of training) and to specify the rates 
or ratings to receive the training (the "who" of training). These factors, 
once determined, finally yield an estimate of the resources required, in 
terms of manpower, equipment, facilities, and money, and indicate when they 
would be required. 

DESCRIPTION OF AND COMMENT ON- THE EXISTING SYSTEM. Analysis and planning 
decisions are made in a wide variety of locations within the NAVEDTRACOM, 
by a wide variety of techniques. Broadly speaking, the "what" decisions 
are based on task analyses, conducted partly by the Bureau of Naval Person- 
nel (BUPERS) and partly by the training command. Task analyses are con- 
cerned with describing skills required in existing ratings classifications, 
rather than with analyzing the broader question of whether the ratings 
classification itself is optimal.- (This latter issue has recently been 
addressed in a larger study concerning a new Naval enlisted personnel 
occupational classification scheme.) Once a task analysis is- available, 
the what of training is determined by noting the required skills identi- 
fied in the analysis. 

^-t, Mm/?nTDApl"^^^''^" ?f stions are decided at a number of levels within 
the NAVEDTRACOM, generally by informal means. There is currently work 
underway to develop decision aids, in the form of computerized planning 
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models, for detailed planning of school work loads and for student schedul- 
ing, but by and large the personnel involved in these problems rely on their 
intuition and experience, on the opinions of professional educators' attached 
to the staff, and the costing analyses developed as part of the individual 
training proposals. Special studies are made on issues which involve sig- 
nificant and semipermanent policy changes, such as the issues of central 
versus decentralized training locations, and still other issues form the 
focus of long range research efforts, such as the question of the optimal 
design of training aids. Some of the decisions discussed here are relatively-^ 
well defined. In determining how many classrooms and instructors are needed 
to instruct a given number of students in a well understood subject, for 
example, there is a good consensus about what is needed. F.or overall plan- 
ning and analysis purposes, then, one or many of the techniques discussed 
for well defined problems may be helpful , depending on the particular situa-- 
tion. Simulation models (e.g., TAEG Report 11-2 [Lindahl and Gardner, 1974]) 
should be extremely useful in scheduling student throughput and resource re- 
quirements for particular training courses. Optimization models of different . 
types could yield useful insights in facilities planning and scheduling. de- 
cisions. In some of the other problems discussed above, however, a real dif- 
ficulty is that the issues are poo^i^'ly defined and understood. In the choice 
of training equipment, for example\ there is general agreement that the train- 
ing .equipment should simulate the actual field situation as closely^as pos- 
sible, but the real question is "how close is close enough?" Is it'worth 
$10,000, for example, to add a sixth degree of freedom to a flight simulator? 
These are issues in which there ts a good deal of disagreement and in which 
experimental data are hard to come by and difficult to interpret. Decision, 
analysis allows one to determine just which sources of uncertainty are the 
most critical for any problem, and thus to indicate what studies or research 
efforts are' most likely to yield useful results. • 

"Who" and "when" decisions^ are not, in general, made within the-NAVED- 
TRACOM. Entrance requirements for the schools are set largely by SUPERS, 
but valid predictive work on student performance would be useful to the 
NAVEDTRACOM in planning, in monitoring staff performance, and in assessing 
the effectiveness of training methodologies. Such knowledge could also be 
useful in recommending entrance criteria to those groups which do make ad- 
mittance decisions. The current separation of admitting and scheduling^ 
responsibilities from the training responsibility seems to conflict both 
with the general management principle of consolidating authority and respon- 
sibility, and with a good deal of civilian educational practices. In train- 
ing contexts, however,- particularly in the military, this separation may 
perform a useful function. Training in the Navy is regarded as a service to 
the fleet rather than an end in itself, and the fleet must have a great deal 
of control if it is to be well serviced. That does not mean, though, that 
the impact of training decisions upon the training .command cannot be conriuni- 
cated. Indeed, in order to make intelligent "who" and "when" decisions 
such impact information is vital. Thus the planning information developed 
by the NAVEDTRACOM should be readily available to outside forces. As in the 
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case of the "how" and "where" decisions, formal predictive models of both 
the decision analytic type and tffe conventional operations research type 
can be extremely helpful, and their use should be investigated. There is 
currently 1 ittle evidence of use of these techniques. 

RECOMMENDATIONS FOR MODIFYING THE ANALYSIS AND PLANNING SYSTEM. Specific 
recommendations to improve decision making In analyzing and planning train- 
ing are: 

, 1. Implement computerized simulation models in planning and forecast- 
ing resource requirements for training operations 

2. Disseminate knowledge of decision analytic techniques to those " 
working with 111 defined, planning problems, and analyze specific planning 
problems with these techniques to illustrate the applicability of the 
method . 

3. Investigate the possibility of using simulation techniques to 
project student performance as a function of student characteristics and 
the type of training being performed, and 

4. Disseminate, as widely as possible, information concerning the 
impact of student characteristics on the training process to those who make 

raining admission decisjons. 

COORDINATE AND CONTROL TRAINING 

^As is the case with any large organization, the NAVEDTRACOM consists" 
of an bomber of organizational entities whose individual efforts must be 
monitore^d and coordinated. The- coordination and control prerogatives of 
management are exercised to some extent by all headquarters and staff 
groups, through the depth and span of control differ greatly, of course, from 
level to leyel. Some members of the NAVEDTRACOM have been assigned, as their 
major responsibilities, a coordination and control function. The largest 
single group doing this work is the staff of Training Program Coordinators 
in CNTECHTRA. mre commonly, however, the coordination and control function 
is exercis„ed as i^art of general' management responsibilities. 



DESCRIPTION OF THeVrESENT SYSTEM. At the CNET level is a staff who are., in 
effect, the duty off^lcers in charge of the actual training effort. All 
training actually im^xlemented by the NAVEDTRACOM is funnelled through CNET 
for endorsement, budgeting and funding, and operational review and control 
at the headquarters lev^l. Most of the problems faced by this staff are 
day-to-day issues; a lot\of "firefighting" goes on, with little .time for 
detailed analysis of the. issues. A good many of the decisions which have 
to be made don't require .d6^;tai led analyses since the proper course of action 
Is fairly obvious. In cases^ where this is not so, the staff generally finds 
some sort of workable comprori^ise between conflicting viewpoints, generally 
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with the aid of the staff 's collective, judgement and experience, but rarely . 
with the aid of formal analytic techniques. The long range. issues that do' - 
exist relate to design of reporting and control systems for the groups which^ 
actually do the . traiafng, and determining methods of implementing certain 
directives relating to priority setting and training methods. For example, 
the CNO has directed that a substantial part of the training load now being 
performed in shore-based scfiools Eie moved to the shipboard environment. A 
number of problems in Implementing this directive are foreseen by the opera- 
tional fleet, who feel that 'they have neither the time nor the resources to 
effectively Increase their current training load. The CNET staff Is concerned 
with determining which portions of the training can be effectively moved to 
the ships and how to make the training work in that environment. 

At the CNTECHTRA level is a group tasked with planning for a'nd allocat- 
ing resources to the training locations. This staff works out contingency 
'plans, such as what would be done by GNTECFfTRA In case of mobilization, and 
operational plans, such as how training should be relocated In the event of 
a facilities shutdown, for approval by CNET. This branch is also heavily 
involved in planning iaterservice training and in determining hov/ portions 
of the training process could be moved to the shipboard environment. The 
staff also has responsibility for resource allocation, including the allo- 
cation of instructor and support billets in CNTECHTRA, and providing physical 
facilities to support training. The branch utilizes various rules of thumb 
on student/teacher ratios, etc., in making these .allocations. 

Other groups in CNTECHTRA have responsibilities for the direct super- 
vision of training activities. These staffs decide how a course will be 
run, what type of instruction will be used, what facilities will be used, 
what the pace of the instijuction will be, and so forth.. The CNTECHTRA 
groups rely heavily on advice and directives from CNET, CNETS, and the 
CNTECHTRA .research groups, but there remains sufficient latitude to signi- 
ficantly affect the efficiency of the training p)?i)cess. 

Most of the detailed design of a course is done by a Training Program 
Coordinator (TPC), an officer or chief petty officer with expertise either 
in the subject area of the training or in the training process itself. The 
CNTECHTRA staffs are responsible for training the. TPC in his job and super- 
vising his efforts, but he usually has a good deal of latitude in determining 
the details of the course. His suggested curriculum, course length, etc. , 
are subject to review and correction by CNTECHTRA, or CNET. 

COMMENTS AND RECOMMENDATIONS. Managers rely heavily on historical methods 
and rules of thumb, so examination and rationalization of those rules would 
be desirable. In determining optimal resource allocations, training schedules 
or type of instruction, to take some specific examples, research efforts are 
worthwhile if results can be communicated to operating managers in a useful 
way. Such communication can take place in at least three ways: 
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• through training and seminar programs and written material, where 
results can be sunimartzed and the effects upon training decisions 
discussed 

• through directives, where results are not discussed but the impli- 
cations are encoded In formal procedures and control mechanisms, 
and • ■ 

• through decision aids, from simple decision rules (e.g. , optimal 
instructor to student ratios) to computerized decision models, 
where the manager specifies* some parameters of^his problem and the 
computer determines the optimal action. 

Which combination of means is chosen depends strongly upon the nature of 
the particular problems being addressed. 

IMPLEMENT TRAINING 

This function includes all efforts- directly involved with the actual 
conduce of training. A number of activities are involved in the running 
of the classroom, such as the preparation and distribution, of instructional 
materials, the conduct of class sessions, and the evaluation of students; 
these are all included In the training implementation function. This func- 
tion also includes such preliminary activities as the preparation of the 
detailed course outline and presentation sequence, determination of specific 
behavioral and knowledge objectives, and training the administrative a>id 
instructional personnel, and such post-training activities as training eval- 
uation and evaluation of testing methods. These activities tie fn closely 
with the course requirements as developed in the "analyze and plan training" 
function discussed above. Finally the "implement training" function includes 
some data management and administrative efforts, such as student enrollment, 
student record maintenance, staff administration, etc. 

The types of decisions «made in this area are detailed, nonquantifiable, 
and generally of too small a .scope to warrant must individual analysis In 
running a classroom it is not possible to explicitly analyze the effect of 
each possible way of teaching a class. There is, or course, a difference 
between good and bad instruction, but good instruction results not entirely 
from good administrative decisions but from good- teaching techniques as well 
A number of broader, analyzable issues are, however, raised in the implemen- 
tation function. Such issues as good general approaches to course material, 
optimal class sizes, and reasonable iiieLhods of training evaluation become 
relevant only in a classroom sltua,tion. 

EVALUATE PERFORMANCE 

J5-^-!'"^*^*^°" ^^^^^ ^""^^ determination of how well training prepares 
the individ^a. to perform his job assignment. Thus, it is a post-training 
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feedback related to the relevancy\of the training itself,- .as opposed to the 
incra-trainlng feedback conducted Vinder the "implement training" function, 
which is designed to indicate how well Individual students are doing in the 
course: Post-training feedback is critical to the overall effectiveness of 
the training effort, since the abserrce of effective feedback froni the func- 
tional Navy could result in training Vhich is inefficient, ineffective, or 
irrelevant. There is no widely accepted "best" way of evaluating training, 
however, as all evaluation .techniques are highly subject to bias and incom- 
plete, superficial, and irrelevant data. Overcoming these problems com- . 
pletely is impossible, and even lessening' them to a significant -degree is 
costly and time consuming. Due to the importance- of the problem, there is 
nevertheless a continuing effort to refirte the techniques and to gather ^ 
feedback information. ' \ 

DESCRIPTION OF AND COMMENT ON THE PRESENT $YSTEM, Feedback* information is' ► 
broken into two major types, summative feedback, which generally indicates 
that a problem exists but which gives little information j?n how the problem 
could be correctec^., and'formati ve feedback, which yields Mjiformation on the 
particular problem and indicates how the problem could be polved. Examples 
of summative feedback are unsolicited letters, material inspections, fleet 
exer^cise reports, operational and technical evalua'tion reports, some 3-M ^ 
reports, 'etc." These documents can indicate problems in the operation and 
maintenance of equipment which can sometimes be traced back to inadequacies 
in the training courses. Formative feedback, which is designed around the ^ 
evaluation of a specific course, involves a determination of the objectives 
of a course then an assessment of whether or noti these objectives are being 
met: I Jo be effective a course must cover the objective material, the train- 
ing must be done in such a way that behavior in field situations will be 
positively affected (i.e., there must be a transfer of the, training) , and 
the exam criteria used in the course must be relevant to field ^situations. 
The technique usually used to obtain this feedback is to ask the man and his' 
supervi'^or for comments after he is back- on the job,* This can be done via' • 
a questionnaire or during visits to the duty station by a NAVEDTRACOM staff 
member^. Evaluation methods of possible use to the NAVEDTRACOM, other than 
those mentioned above, are interviews with rotating fleet and training"' per- 
sonnel, experimental methods, use of performance diaries, and formal, struc- 
tured Visits to duty stations, ^Each af these methods, along with their 
advantages and disadvantages , are mentioned briefly below. A fuller dis- 
cussion can be found in TAEG Report 12-1, Vol 1, pp. VI-190 and 191. 

' In evaluating, the re^suTts of any feedback method it must be rem'embered 
that the fleet often does not understand the purpose of a given training 
course, thus is prone to giving erroneous feedback, and that the NAVEDTRACOM 
has little control over the qualifications of incoming trainees, ^ The seeming 
inadequacy of a training course may not be due to the course desi-gn or im-^ 
plementation at -all, but due to the lack of qualifications of the trainee,' 
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^' From a theoretical point of view the evaluation picture is not tidy. 
Efforts are fragmented and tend to be superficial. Before discussing ways 
of improving this, however, it is worth considering how much effort the 
evaluation function is worth. Evaluation can be extremely demanding of 
money, time, and personnel. Specialized' skills, which are generally scarce, 
are required. . Clearly some reasonable level of evaluation effort must be 
determined, and that level may well, be less than the theoretical "best that 
money can buy." In addition, one must recognize that training does not occur 
in a, vacuum. Training goals and course objectives are set in response ^to 
requests from and in conjunction with the operating Navy, The NAVhDTRACOM 
staff is generally familiar with Naval needs and procedures; trainees do go 
out to duty stations and occasionally word on performance does get back to 
the. training command, all without too much effort on the part of the NAVED- ' 
TRACOM. Biased, sporadic-, and distorted as these informal means are, they 
sum to a fairly large amount of useful information flow. The next level of ^ 
effort, which involves such active but relatively simplistic information 
search techniques as mail out questionnaires, infor^mal visits, tq duty sta- 
tions, and s,creening of existing data ban'ks, is alsa in wide use in the 
NAVEDTRACOM. Again the total amount of information gained through such 
means is consequential. Such techniques would be grosslj^ inadequate for a / 
research oriented institution, but for an organization providing more-or- 
less standard kinds of training in inore-or-less standard ways they may 
suffice. This is not to say that present methods could not be improved. 
Present questionnaire design leaves much to be desired, and the timing of 
the questionnaire mailings may not be optimal. The limitations of informal 
feedback tend to be Ignored. There are a number of cases where cooperatiion 
and information exchange among different schools would be useful, but thajre 
is no integrated effort to obtain feedback. (See TAEG Report 12-1, Vol 1, 
pp. VI-224 and 225 for a discussion of these points.) Finally, there appears 
to be a legitimate, though limited role for the more sophisticated techni- 
ques of evaluation, namely use of professionally designed and evaluated 
experiments, use of performance diaries, and" other structured performance 
records,. and use of structured interviews. A capacity in this area would", 
enable in-depth analysis of new teaching methods, evaluations of problem- 
courses, and be a good source of knowledge for the educational process in 
general. 

<? 

Once a system for gathering , feedback information in an effective manner 
is established, one must be concerned with how. to determine measures of 
training effectiveness. This again is an extremely difficult research area 
as there is little agreement on ^hat, 'specifically, constitutes effective 
training. It is- possible to gather experimental data on, say, how often 
given tasks can be performed successfully, how quickly students can respond 
to simL(lated emergency conditions, how 'well students have mastered factual 
data, etc. Further, such data can be gathered and compared for different 
trainirig methods, enabling one to obtain some relatively objective effective- 
ness comparisons among the methods. With proper experimental design, the 
effects of inherent differences among students could also be negated. Un- 
resolved problems, though, exist in the following areas: 
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1. The above approach yields many measures of effectiveness in different 
tasks, while a single overall measure is desired, 

.2. The individual data items gathered must relate directly to effec- 
tiveness in operations, and there are difficulties in establishing this 
relationship, 

3. ■ Experimental data are often difficult to gather and subjective in 
nature (what, for example, constitutes successful performance of a task?), 
and 

4. Experimentation is expensive, and it Is not clear that experimental 
establishment of measures of effectiveness is always an effective use of funds. 

Although these problems must, to a large extent, be dealt with through infor- 
mal techniques (experimental ingenuity, understanding of job requirements, 
experience), formal techniques could be very useful, particularly in combin- 
ing multiple measures into a single measure of effectiveness, in clarifying 
assumptions, and in evaluating the potential worth of experimentation. 

RECOMMENDATIONS FOR CHANGING THE PRESENT SYSTEM. What is' needed is first, an 
Tmpfoved method of collecting data .and, second, better V)/ays of determining 
total measures of effectiveness. Consider first an improved data collection 
system. Such a system should involve a series of effort levels; first the 
'•no effort" methods can generate rough information on many areas; second, the', 
"moderate effort" methods can be used to generate data on specific courses 
and identify problem areas; and third, "high effort" methods can be used to 
study problem areas in de]3th and perform research in the learning process. 
A number. of specific suggestions are included in TAEG Report 12-1, pages 
VI -224 to VI-235. The "measures of effectiveness", problem is basically a 
problem involving, multiple measures of effectiveness under uncertainty. As 
discussed in section' II. B, pageM7, multidimensional utility theory can some- 
times be used to determine a single measure of effectiveness and expectedNutil i 
used as a decision criterion under. uncertainty. This approach also indicates 
promising areas for further research. It is reconinended that those persons in- 
volved in research in these problems be, famil iarized with -these techniques. 

MANAGE RESOURCES 

This function encompasses the management techniques which are associated 
with resource control in any organization. Some of these are: (1) forecast- 
ing expenses and constructing and defending budgets; (2) monitoring expen- 
ditures of resources vs budgeted amounts , then- correcting and updating bud- 
gets, and implementing corrective action' to control budget variances; 
(3) material management, including providing, maintaining, supporting, and 
controlling material and equipment, quality control, maintenance and logistic 
support; and (4) the allocation, utilization, and record keeping of personnel 
within the training coninand. In the NAVEDTRACOM these actions take place 
within the context of the standard government planning and budgeting cycles. 
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which enable the CNET^budgettng system to interface with Navy, DOD and 
Congressional budgeting data. Procedures .for these efforts are spelled out 
and documented \n some detail in standard government directives. Some oara- 
^^A^rlrf ^^f organizational design are also fixed by outside authority, 
^nd CNET must -operate within those limits. ' "Muy. . ^ 

IIlksCRIPTION OF AND COMMENT ON THE EXISTING SYSTEM. Some of the decisions 
categorized under resource management are highly routine. Some of these 
lo'Jnnlf .nlwJc^ inventory policies for spare parts, are highly amenable 
Scin 1 analysis; industrial experience has -shown that careful analytical 
design of inventory systems, to remain with this particular case, can be 
most rewarding. A number of other routine functions, such' as record main- 
tenance, are primarily administrative rather than decision oriented in ° 
nstuurG . 

other decisions may seem routine, but are critical to the success of - 
the training effort. Allocation of people within the staff, for example 
IS an important determinate of effectiveness, efficiency, and morale. Selec- 
ThrMfllFnTSJmr'I' te^can be critical in the. success of an effort. 

The NAVEDTRACOM staffs rely heavily on experience and other informal mecha- 
msms for making these decisions. There are also groups tasked with provid- 
ing budgeting and financial expertise, acting primarily in advisory roles 
to the rest of the command, projecting the resource needs of various pro- 
posals, outlining funding alternatives and interpreting budgetary constraints 
for the other groups. These staffs are generally represented in decision, 
making groups, but in the role of defining feasible alternatives and per- 
forming economic analyses of decision alternatives, rather than in the role 
of advocating particular projects. Such practices are in agreement with 
usual management techniques, and cannot, in general, be faulted. Detailed 
invpstigation of specific decision practices in this area was not.under- ' 
taken in this research.- g ci . 

RECOMMENDATIONS TOR IMPROVING THE CURRfNT SYSTEM. In those areas of resource 
management which involve well defined, routine functions, such as inventory 
management, production control, quality control, forecasting demand, and 
other areas, use should be made of the wide variety of analytic techniques 
cormonly used in modern managemen.t. Industrial practice has demonstrated 
of such Sy sis improvements can result from proper application ^ 

PERFORM RESEARCH 

The_"perform research" function includes all activities concerned with 
the acquisition of knowledge relating to the training, learning, and evalu- 
ative processes. As used here, then, the term "research" includes not only 
research in the sense of conducting formalized experimental situations, but 
field studies and surveys, statistical analyses, ^nd monitoring ongoing 
research m other locations as well. "Research" can mean either bas c or 
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applied studies. The NAVEDTRACOM conducts some research in-house, sponsors . 
some research performed by outside contractors, and reviews a good deal of 
work performed by the research community at large. 

DESCRIPTION OF AND COMMENT ON THE PRESENT SYSTEM. Research groups within 
, the training command have the responsibility of identifying current and 
future needs of CNET, then insuring that research needed to satisfy those 
needs is performed. This involves: first, .monitoring all relevant research 
and development efforts, both within and outside the Navy, and identifying 
results of interest to CNET; second, coordinating efforts among research 
groups whenever appropriate; third, determining priorities among potential 
research efforts which CNET could encourage or sponsor; and fourth, designing 
and conducting or sponsoring specific research projects. The first two 
/fefforts are administrative rather than decision oriented. Some groups are 

/tasked with the role of being a research broker. Researchers and possible 
funding sources are brought together. In return the NAVEDTRACOM asks for 
support for its own research goals. The third effort involves the need for 
a powerful decision making capacity, and being a prioritization function, 
r it shares many of the needs 'of prioritization schemes as discussed in section 
V III.B. The research prioritization function differs from the training prio- 
\ ritization function in that uncertainties are even greater and objectives 
\ lessN^ell defined in the research prioritization. Finally, a good deal of 
Veseafch is done in-house by the training comiiand. Some of this is very 
spBisiftc, such as research into the teaching process. Task analyses are 
perfonrled for given, jobs, alternate methods of teaching tasks are considered, 
human factors analyses are performed to determine the most promising teach- 
ing methods, and programs are evaluated. For example, such research at the 
CNTECHTRA level is oriented, toward very specific criteria, such as lowering 
student attrition rate, optimizing use of instructor time, or minimizing 
costs. Other efforts are directed towards mere general problems, such as 
methods of determining optimal equipment designs, ways of measuring. the 
transfer of '^training, and so forth. 

RECOMMENDATIONS FOR IMPROVEMENT. Comments on optimal experimental design 
are not made as that is tooilarge a subject to address here. The subject . 
of choosing which data are n]ost worth experimental pursuit is, however, 
•essentially a prioritization problem. The techniques discussed earlier 
should be just as beneficial in this setting as in project prioritization, 
thus it is recommended that managers charged with prioritizing research 
. projects investigate the possibility of developing, through decision analytic 
techniques, a formal ranking scheme. to account for quantifiable factors. 
Nonquantifiable factors could be handled through prioritization committees 
and individual judgement, as fs being done now. 

SUPPORT TRAINING. 

This function includes all activities directly relating to the support 
of training operations. Among the specific items accomplished are: 
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1. Development, procurement, and maintenance of training equipment, 
ranging in scope from small- inexpensive items to room-sized -simulators, 

2. Development and dissemination of self-instruction materials and 
training packages for use in on board training, 

3. Procurement and dissemination of books for libraries, 

4. - Development, administration, and scoring of tests for advancements 

in rating, and - ^ 

5.. Recommending course structures and training media. 

DESCRIPTION OF AND COMMENTS ON THE PRESENT SYSTEM. The first area of respon-^ 
sibility of the support staff concerns equipment design and selection. 
There are two types of decision tasks, fal ling in this area. First, in de- 
signing new equipment,, trade offs must be made among cost, schedule, and 
various performance parameters. The second, related task involves both 
monitoring and evaluating .new developments, both technical and educational, 
and determining when to "freeze" equipment design. Sometimes these tasks 
are identi cal , such as in determining the design for a new one-of-a-kind, 
high cost simulator. In other cases these are distinct types of decisions. 
In providing operating units with such standard equipment as projectors and 
tape recorders, for example, CNETS must continually balance the benefits 
of standardizing, on newer, improved equipment versus the extra costs of 
obsoleting present inventories and spares and restocking a logistics system 
for the new unit. Decisions in this field are difficult to make, but. both 
these types of issues can be approached "more systematically through decision 
analysis than is done traditionally. 

To illustrate the .general approach to analyzing these problems, consider 
an examp-le in equipment design. This example is grossly over simplified, 
but it should serve to outline the technique. 

Suppose a large one-of-a-kind simulator is being designed, and it is 
unclear whether or not a particular feature, costing $D, is worth including. 
The key issues, clearly are (1) how much better, or faster, or cheaper, 
will training be with the feature, and (2) how much, in dollar terms, is 
it worth to have better, faster, or cheaper training. Let us first ddscuss 
the dollars issue. Sometimes ,, as is the case when the feature leads to 
cheaper training, any theoretical difficulties go away. A severe practical 
problem may exist in estimating the magnitude and duration of the savings, 
but one could easily use a simulation model to determine- how much time and 
effort is warranted in collecting that data. Likewise, if the feature leads 
to faster training it should.be possible to simplify the problem by estimat- 
ing the cost per day of training," then translating the time savings to a 
cost savings and analyzing as above. If the feature leads to an improvement 
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in training (i.e., an increase in skill level), it would be necessary to 
quantify skill' level in terms of one or more measures, then determine trade 
offs among cost and skill levels. This can be done through multidimensional 
utility theory. ■ , ' \ 

A more difficult datum to establish is the degree to which training 
will be improved by the feature. Not only is the training mechanism poorly ^ 
understood, but results should vary from student to student. One can,, 
however, use decision analysis to quantify the degree of uncertainty.^ , 
People expert in the field could be, asked to assess probability distributions / 
over the range of possible improvements, then .this data can be used to deter- / 
mine whether a decision can be made immediately or whether experimental ^ 
evidence of various kinds should be gathered. Even in highly uncertain / 
areas it is usually remarkable how closely experts will agree on the limits j 
of their uncertainty. , / 

The second area of responsibility in training support includes writing 
and updating training manuals and texts. In writing manuals the selection/ 
and structuring of materials is clearly of major importance, and careful 
coordination between the support staff and outside groups is necessary to^ 
insure ah effective product. 

The third area of responsibility, supporting general libraries afWat 
and ashore, is not so much an issue in itself, but does ^^^e once more/ 1^^^^ 
problem of project prioritization since library support functions must/com- 
pete with other areas for funding, and it is difficult to quantify the/ bene- 
fits of supporting libraries. As is generally the case within the training 
command, there are more projects coming to the support groups than caf_ be 
handled, so some method of prioritization is needed. It was found that 
support groups rely exclusively on informal schemes, such as the priorities 
board. A number of precedents exist to guide funding decisions on venous 
types of efforts; these are supplemented by staff studies on pmject validity, 
benefits, and costs, and the collective experience of the board. Also, data 
from the field is generally collected on the projected impact of funding 
alterations. In determining the effect of investing a given amount of money 
into simulators versus libraries, for example, both internal and externa 
opinions are solicited on the probable effects on the quality of the train- 
ing effort, but no attempt is made to formally measure or trade off those 
eff'^cts. If all relevant issues in these decisions are indeed nonquanti- 
fiable, this may be the only possible way to proceed. .Almost always, however, 
it is possible to systematize at least part of this process and to improve 
the quality of decision making. • 

The fourth area of responsibility, relating to testing training course 
gradu^'.tes, is a significant aspect of the support function, but is not a 
significant decision activity. Thus, it is not discussed in this report. 



57 GO 



TAEG Report No. p 



The final area of responsibility conce/ns training structure and oraani 
.jat.on. The support staff supports this e/fort with task and course analjs 

q i ra?ninT'\''ann7H'1''"? '^'^''^r' °" °P^^'"^^^ ^^^^ restrucS- 
JlAil llA ^'- ^ °^ "^^"^ 0" recommending training 

media and equipment appropriate for give/ courses. 

?LTraiI?.p/L' r'^™;.-'' /difficult to make blanket recommenda 
r MH?nn H • ^^'^ ''°^'"3l decision techniques, in- 

dS? cuu'Jo'Jp? nn'^/'"'; *° h^ve potential usefulness, btjt it seems 

difficult to set up a system to use such techniques on a routine basis 
Perhaps the best way of introducing support groups to the potential of'for- 
Serna "cSnlulJ.'n?." 'll' ' relatively small group then le'f them act as 
n Sd fadl"ua"e"uc;'a'Se^ """^'^"^ '''' ^^-^^^ established, 

ADMINISTRATION AND INTERNAL CONTROL 

<:nM./?''^^""?*^'°" ^■"^"'^^es personnel administration; a number of human re- 
sources, equal opportunity, and drug/alcohol programs; such management 

:nar.ed"?„^;hi^"?fpS??f -^^"^ these .rJ\T.T 

MISCELLANEOUS FUNCTIONS 

<^nn^ The training command performs a variety of miscellaneous missions'. 

ar^. P^ntnLTf'-' ^^''^^ ^^'^^ble, are educational pro- 

grams, equipment design, and budget advice. The educational programs area 
includes administration of, curricula development for, and seleS of stu- 
dents for the Navy Campus for Achievement, the NROTC and Junior NROTC p^-o- 
grams the Nava Postgraduate School, the Naval- Academy, special or graXte 
education in^un versities, officer enlistment programs, those overseas 
dependent schools for which the Navy has been assigned responsibility! Jnd 

recoanUirth'TIh' '^^^'l^'"?-, ^^"^'P^^^ ^^e.a ar'ises from the 

recognition that the initial design of a piece of equipment clearly has a 
bearing on the amount and type of training later required to qual fy per- 
.n pfL^2-°''T*? '""^ '"'][^*'!" '^^^ ^^^^■"^■"g tries to establish 
.n flit fl'^ dialogue with the equipment design office as early as possible 

CNET dLrnn?'n.^""''^r'*'°"' ^"ti ci pated . , In this relatibnsh p 

CNET does not act as a decision maker, but as an advisor. The budget 

asTt°affPr^ '° t'S' governmental budgeting and accounting'cycle 

as It affects training. The training 'conmand works with the operating Navv 
in projecting^training^needs and including required resources inlraining 
budget. requests. In addition, cooperation and advice is given in arranSinq ' 
funding for items submitted too late to include in budget estimates 
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C. 



GENERAL OBSERVATIONS ON DECISION MAKING IN THE NAVEDTRACOM 



Decision making within the NAVEDTRACOM reliesi heavily upon informal 
judgements, group consensus, rules of thumb, and traditional practice. 
This is a common mode of decision making in large organizations and is . ' • 
rather to be expected. In the NAVEDTRACOM, in factu pressures toward such 
behavior appear to be even greater :than in corporations. 

THE NAVEDTRA.COM DECISION ENVIRONMENT \ 

Decision problems which are easiest to solve through analysis are those 
which are well defined, where uncertainty is negligible, and where a single 
decision criterion (measure of effectiveness) exists.] Most problems in the 
business world are not this tidy, and in the NAVEDTRACOM the vast majority 
of decisions are even- more complex than in business, ^hile a business can 
usually use financial gain criteria at least as the starting point of an . 
analysis, in the training command this common measure i\s lacking. The "cost" 
side of training is. readily apparent, but the "benefit"^ side is nearly un- 
measurable.. One cannot easily put a dollar value on the benefits of training, 
so the best one can hope to do is to determine the minimum cost of meeting 
preset 'training criteria. When it comes to deciding what those criteria 
should be, howeverj a subjective evaluation must be made. In addition, one 
must recognize that the NAVEDTRACOM decision maker is not only a part of the 
training command, but a member of the Navy. The tightness of the Naval 
community, the importance of personal influence and contacts, the permanence 
and importance of the fitness reporting system, and the ever present concern 
with one's career all combine to give "political" considerations extraordi- 
nary import ance. One always considers not only how good or bad a program 
looks on paper, but who is pushing it and how much weight he carries. Such 
pressures exist, of course, in every large organization, but in the military 
they seem to be particularly evident. This tends to confuse the effective- 
ness measurement picture even more, as a whole list of new factors are 
brought into every decision. ^ 

Another critical factor in the decision environment is the fact that 
the military manager is a highly transient individual. The frequent rota- 
tion cycle encourages a certain' shortsightedness in Naval management methods , 
one consequence of which is that out-of-the-ordinary "things , -especially if 
they are complicated, do not get done. In industry a good many analytic 
efforts start with the "conversion" of a manager, who then uses the method 
in his own department, on his ov/n problems. He builds up an e^xpertise and 
a staff familiar with the method and produces, eventually, some concrete 
benefits, in the Navy it is difficult to do this, as both the^- manager and ♦ 
the staff are continually shifted. There is little incentive to try some- 
thing new, no matter how promising, unless the payback is immeaiate. 
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SUMMARY OF RECOMMENDATIONS FOR IMPROVING DECISIONS IN THE NAVEDTftACOM 

Given the limits of this environment there still appear to be promising 
areas for improving the decisions made by the NAVEDTRACOM. Further the 
magrntude, importance, and complexity of the problems being faced make better 
decision making techniques imperative. In this research a number of oppor- 
tunities for improving decision making have been identified and discussed 
buggestions can be classified as being improvements in organization, in data 
gathering procedures, in decision rules, in analytic capability, and in 
analytic techniques. 

\ ■ 

Organizational improvements refer both to formal organizational changes 
and to Changes in formal communication mechanisms. If separate groups each 
possess only portions of the infonnati on necessary to make rational deci- 
sions., then cooperation and information exchange among the groups will often 
. I.ipprove the quality of decisions made. This does not happen automatically, 
^ince proper reward structures and task definitions must also exist, but 
complete discussion of these issues would go beyond the limits of this report, 
bpecitic of^garnzational improvements that can be suggested are: 

• ther role of prioritization committees and other management groups 
in project prioritization be modified to emphasize only the non- 
quantitative factors, with quantifiable factors being included 

in a formalized scheme, 

• in developing training requirements, the NAVEDTRACOM and the 
operating forces should set up a joint, continuing effort to de- 
fine and project requirements, ' 

• CNET should disseminate projections of 1:raining costs and success 
rates as a function of student characteristics, and 

• in training support, continuous communication between the staffs 
involved in writing training manuals and texts and the schools is 
essential . 

The_professional background of the researcher precluded an exhaustive exami- 
nation of organizational issues; undoubtedly, much more could be said in 
these areas. 

Data gathering pruDcedures can be improved particularly in the training 
evaluation function. A range of feedback techniques should be used to deter- 
mine weaknesses in the current training system. In addition, techniques of 
applying some o« the methods currently used can be improved, aj discussed in 
TAEG Report 12-1, pp. VI-224 to VI-235. . 
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Decision rules are defined as the specific guidelines and procedures 
by which organizations make decisions. Often these are very informal, being 
expressed as general rules of thumb, or they can be formalized as standard 
procedures. Sometimes there is a good deal of wisdom built into these rules, 
but sometimes they lead to non-optimal decisions. Thus, a systematic, 
rational redefinition of decision rule 5 may pay dividends,' particularly 
since it is usually possible for an operating manager to apply the rules 
without understanding the details of the analysis behind the derivation 
of them. Decision rules can be expressed in the form of simple "if-then" . 
statements ("If anticipated training load is greater than 100 students per 
year then set up a new course"), in the form of tables and charts which are 
used in scoring possible actions, or in the form of computer models. Some 
areas where rationalization of decision rules seem particularly appropriate 
are: 

• in the formal phases of project prioritization, whereby project 
characteristics can be used to derive a priority ranking much as 
is now done in the training requirements area, but with the tables 
being more systematically derived, 

• in the analysis and planning area, where computerized planning and 
forecasting models can usefully capture a good part of the com- 
plexity inherent in such situations, and 

• in the coordination and control area, where many day-to-day deci- 
sions on training administration must be made on an ad hoc basis, 
but where a' systematic examination of such issues as optimal 
student/instructor ratios^ optimal methods of presenting various 
types of material, etc., could be beneficial. 

Improvements in the analytic capability of an organization can be brought 
about by increasing the level of analysis which underlies standard decision 
rules, as discussed above, by using ou.tside specialists in the analysis of 
particular problems,, by establishing groups of analytical specialists to be 
used as internal consultants, and by increasing the general managers' under- 
standing of and use of analytic techniques. Each of these approaches has 
its particular strengths. Through improving the decision rules one can 
parsimoniously improve the mass of decisions, but unusual or extremely im- 
portant decisions cannot be handled with standard rules." Outside experts 
can bring particularly needed expertise to bear and can be hired only as 
long as needed, but they can be expensive; they must become educated in the 
organization's problems and must be found and identified. Internal consul- 
tant groups are invaluable for longer term expertise and are usually familiar, 
with the organization, its problems, and often its personnel, but may lack 
expertise in specialized areas. Increasing the analytical sophistication 
of the general manager is difficult and expensive, particularly in highly 
transient organizations, but some education is necessary before the manager 
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even recognizes tlie possiblity of formal analysis\ All -four approaches are 
thus appropriate for different circumstances, and k balanced program is 
necessary to derive maximum benefits from formal ana\lysis. 

A few of thk^ecific areas in which an enhanced\analytic capability 
would be useful are in the determination of the relationship of student ' 
characteristics to length, proficiency attained, and drdpout rate in training- 
in the planning of training, where. a number of recurring\decision problems 
regarding training structure occuf; and in the resource management function. 
Which is an area where formal techniques have returned subkantial bpnefits 
in industrial settings. • \ ' 

T JJ^-^ii^^ suggestion involves improving analytic techniques. As out- 
lined briefly in the taxonomy-, a number of analytic approache^have been 
developed for dealing with decision problems. A fairly new a nd\ extremely 
powerful series of techniques, decision and utility analyses,- prVise an 
approach to solving problems which are intractable by any other i^eans. 
Analytic expertise in this field is valuable in determining measures of 
etfectiyeness, in dealing- with problems involving uncertainty, in Working 
L^^''"^^ problems, and in working with problems where results are 
difficult to estimate or quantify. ' \ 

_ A specific suggestion arising from the last two paragraphs is to\stab- .. 
Iish an internal consultant group familiar with formal analysis, particularly 
TDJrnM°" ^'?3'ysis. Some consultant type. groups already exist in the NA^CD- 
IKACOM Since these groups already exist as operational entities and al\ 
ready have a good deal of analytical expertise and experience, adding a dh- 
cision analytic capability would not be too difficult. This could be done^x " 
in a variety of ways. The Naval Postgraduate School is able to offer educa^ 
tiona I programs, from 'brief introductory courses to accredited short courses 
■?ho MflvpnTJflroM. ^!]^»^^ is interest in doing this, a program tailored to \ 
the NAVEDTRACOM's needs and desires can be arranged. Self-education is a 
second possibility. The classic introductory text in decision analysis is \ 
Decision Anal ysjs by Howard Raiffa, Addison-Wesley, Reading, Mass., 1968 
A newer text, perhaps more appropriate for the general ist, is Decision' 
J^iLZSIs. for the by Brown, Kahr, and Peterson, published by Holt, 

Rhinehart and Winston, New York, 1974. The Naval Postgraduate School has 
a capability in developing programmed learning material, and could be tasked 

fn o^lll^^yi rll^T^"^^ decision- analysis. A third possibility is 

to educate CNET personnel in university courses in decision analysis A 
T^ri?L°^ '?°?^' offer such courses. The two most well known being'the 
graduate schools of business administration of Harvard and Stanford Univer- 
ViVf'A- I?"""?^ possibility is an expert in decision analysis could be 
hired directly to an internal consulting group. This is clearly the most 
?nncM?ffi °l gsininc expertise. A final suggestion is to hire outside 
^o^c ^ J. Jor selected analysis projects. This would not only altow mem- 
bers of the training command to see the techniques in practice, but W 
Mcfri-^ J usefulness of the approach. External, consultants can bIso be : • 
used directly in educational roles. 
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SECTION IV 

SUMMARY AND RECOMMENDATIONS ■ 

In this section, the major points and concl'usions reached throughout 
this report are summarized, some suggestions for implementing change given, 
and recommendations for further research and development work made. 

A. SUMMARY OF THE REPORT 

In section II, the background of the research is presented. Organiza-' 
tional decision making is discussed in section II.. A, the major observation 
being that organizations typically rely heavily on "irrational" techniques 
for decision making, but that there are very. rational reasons for doing so. 
Then in sections II. B and II. C, some background in formal, rational tech- 
niques of decision making is given, with section II. B containing a taxonomy 
of decision techniques and section II. C an outline -of formal decision 
theory. These two lines of thought are brought together in section II.D, , 
where the actual and potential use of ratiqiial decision techniques in pri- 
marily irrational decision environments 'i^s-^discussed. It is pointed out 
that decision analysis not only, shows. great promise on paper, but has 
demonstrated significant benefits in actual situations. The attitude of 
the manager and the competence of the analyst, however, are critical to the 
success of any analysis. f 

In section III, a broad overview of the. functions "performed by the 
NAVEDTRACOM is given, then the types of decisions made within the command 
are discussed in some detail. Section III.C summarizes observations of 
NAVEDTRACOM decision making, pointing out the extraordinarily heavy . 
reliance on informal techniques for critical decisions and indicating some 
reasons for this. A number of potential areas for improving decisions are 
identified and the reasons for considering these areas are discussed. 

. B. IMPLEMENTATION STRATEGIES 

There are two routes to be considered in improving decisions in an 
organization— a "cookbook" approach and an "educational/consulting" ap- 
proach. In the "cookbook" approach the decision maker is given a set of 
instructions which he mor,e or .less blindly follows in order to reach his 
decisions. This method has a number of advantages: a good deal of effort 
and intelligence can go into designing the instructions, insuring a well 
thought out and complete decision making procedure; the effects of person- 
nel rotations on decision making are minimal; the instructions can be 
gradually improved as experience is gained; the instructions can be ex- 
pressed in many different forms, such as directives, standard operating 
procedures, decision tables, computer models, or informal recommendations; 
anduse of standard instructions leads to consistent, predictable decision 
making. In some situations, such as in computerized systems,' some simple 
instructions may by necessity be built-in. Even in situations where 
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1. Use of supporting relationships: This is exemolified hv ^ mncm 
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. 3. Articles and written summaries: a good deal of writtpn mafPK>i;,i 
a" "'l„"^he1?:?d'n?'r'-'' techniques exists, sole "oV^lliJ^^-f Ju te' 
Rev ™ ha? ^ J*lf ^e<:'=l°n analysis, for example, the Harvard Business 
^HrfaS?^iTt?s?rana?;?l!^^'" ''"""'"^ only M-ooTbiF 



4, Textbooks: A number of 
go deeper into analytical 



_ books, including some programmed texts 
techmq^es than an article could do. 



three ^dav.^rn.nn'L ^'^ ""e^gth from a half day to two or 

of lifted s?ze '^'^"^ ' '^''^^ introduction to a group 

6. Short course: The Naval Postgraduate School is currpntiv nffo^-inn 
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First, J* n working toward effective implementation, it is recommended 
that a group (like TAES) which is already involved in analysis and con- 
sulting within the NAVEDTRACOM^ be identified and given expertfse in 
decision' analytic techniques, the group is likely to be more receptive 
to new techniques than general management, and having already established 
relationships within the command, be more effective as an ongoing educa- 
tional agent than an outsider could be. Anr^nternal group would also be 
aware of outstanding problems, and be able to demonstrate concrete bene- 
fits of decision analysis on those problems on which it consults. The 
expertise could be ffi.ast readily developed, throucjh training seminars or 
short courses, either of which the Naval Postgraduate School could offer. 

A second recommendation, which could be accomplished in conjunction 
with the- first, is to identify and analyze a "cookboqk'' type project, 
such as developing a prioritization rating scheme or formulating decision 
rules of a computerized system. 

/ ly^niplementation of the results of such an analysis is less dependent 
oh the manager than for less structured problems. Ideally, this would be 
done: by an internal consultant enrolled in the training program mentioned 
above in conjunction with an outside expert. 

■ • • . • _ . ■ 

The major area of potential improvement Vies in educating the general 
manager. As discussed above, there are a number of methods of reaching 
him. Such an effort should be undertaken in conjunction with establishing 
expertise in an internal group, so the managers could obtain help on 
problen]^ when necessary. General management education should be tied in 
with concrete results on-a real problem in order to lend credibility to 
the technique.' , 

Further research in decision making within NAVEDTRACOM should be - 
more narrow and more intensive in nature than was this .vork. One parti- 
cular organizational unit or function area should be selected and studied 
in depth. Such a study should focus on the (1) types of problems faced, 
(2) frequency of occurrence of each type, (3) decision methods used,' 
(4) pressures and influences which limit decision* al ternati^ves , (Ei) impact 
of the decisions mad^ on the rest of the organization, and (&).^formal and'^ 
informal roles of the organizational unit in the total organi zatipn. ^ * 
Alternatively a single type of problem (e.g., prioritization) couldNbe 
chosen for intensive study, with the same types of questions being asked. 

The NAVEDTRACOM, like any large organization, 1s complex and unwiefdy. 
Its members are subject to human frailties and prejudices, and in the fac6. 
of a complicated and* uncertain world make mistakes in judgment. Decision \ 
analysis, and other formal techniques cannot overcome human frailty, but 
they dd allow one to examine his 'thinking and assumptions in a way which 
minimizes the faults attendant to being human. 
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